Flink Forward Asia

Modern data environments require architectures that seamlessly blend the flexibility of data lakes with the performance characteristics of traditional data warehouses. As enterprises increasingly adopt real-time analytics to drive business decisions, the combination of #ApacheFlink as a stream processing engine with #ApachePaimon as a lake storage format has emerged as a compelling solution for building powerful real-time lakehouse platforms. At the Apache CommunityOverCode Asia 2025, Xuannan Su, Alibaba Cloud Technical Expert and Apache Flink Committer, shared profound insights into the continuous evolution of Flink real-time lakehouse solutions built on Paimon. This technical deep-dive explores key optimizations and architectural improvements developed to address real-world challenges encountered in implementing large-scale streaming analytics platforms. As the volume of structured and semi-structured data grows, traditional data processing approaches often struggle with performance, cost-efficiency, and operational complexity. The discussed enhancements represent production-tested, practical solutions for organizations seeking to modernize their data infrastructure, providing a clear implementation path for scalable, real-time data pipelines. Dive into the full blog to learn more about these cutting-edge solutions! https://lnkd.in/giTmnsFM #ApacheFlink #ApachePaimon #RealTimeAnalytics #DataLakehouse #FlinkForward #AI

At Flink Forward Barcelona 2025, we announced a major collaboration between Alibaba Cloud, Ververica, Confluent, and LinkedIn—four influential companies in the $100 billion data streaming industry—to jointly develop and contribute to Apache Flink Agents, a new open-source sub-project from the Apache Flink community designed to bring AI agents into the world of real-time, event-driven systems. This initiative marks a pivotal step toward industrial-scale AI applications that react instantly and autonomously to live data streams. Apache Flink Agents, a brand-new sub-project from the #ApacheFlink community, is an open-source framework for building event-driven agents. Building on Flink's battle-tested streaming engine, Apache Flink Agents inherits distributed, at-scale, fault-tolerant structured data processing and mature state management, and adds first-class abstractions for agentic AI building blocks and functionalities - large language models (LLMs), prompts, tools memory, dynamic orchestration, observability, and more. This initiative is the result of a community-based joint effort by developers from Alibaba Cloud, Ververica | Original creators of Apache Flink®, Confluent, and LinkedIn, a group of engineers with deep expertise in large-scale stream processing and real-time AI. By combining our experience in production-grade data infrastructure and intelligent systems, we are aligning on a shared vision: bringing agentic AI into the streaming data ecosystem, where it can operate with scalability, reliability, and real-time responsiveness. To learn more, visit the full announcement blog post: https://lnkd.in/gN_GadCc Read the official release announcement for Apache Flink Agents 0.1.0:https://lnkd.in/gP5BBjQW

TikTok is revolutionizing its recommendation system with a unified Lakehouse architecture powered by #ApachePaimon! 🚀 This new approach optimizes large-scale recommendation models (LRMs) that prioritize user behavior sequences, addressing key challenges in data pipeline efficiency, consistency, and scalability. The transition from traditional deep learning models to LRMs has simplified feature engineering and enabled real-time personalization. The unified Lakehouse design includes a four-layer architecture: DIM, DWD, DWS, and ADS, each serving specific roles in data processing and feature engineering. This setup ensures both flexibility and performance, making it a robust solution for managing user behavior data. Read the full blog to learn more about TikTok's innovative data infrastructure and how it's setting a new benchmark for modern AI-driven platforms: https://lnkd.in/gAG2-t2h

The countdown is on! 👀 In just 2 weeks, the Flink community will gather at Flink Forward Barcelona 2025! Join us for two days of hands-on Apache Flink learning, 🐿️ followed by two days of conference sessions and networking. This is where streaming innovation happens! Don’t miss your chance to connect with the best in the business! 🎟️ Secure your spot for Flink Forward Barcelona 2025 now https://hubs.li/Q03JYwB40 #FlinkForward #ApacheFlink

Apache Flink CDC 3.5.0 is here! New pipeline connectors for Apache Fluss (Incubating) (sink) and PostgreSQL (source), plus major improvements in schema evolution & multi-table sync. Fixed key issues in transform, incremental sources, and added enhanced support for Paimon, MySQL, PostgreSQL & OceanBase CDC. 👉 Get it now: https://lnkd.in/gEuhg-zk 📚 Announcement: https://lnkd.in/gYPGNNwJ Huge thanks to all contributors! #FlinkCDC #ApacheFlink #DataIntegration

Grab, Southeast Asia's leading super-app, leverages #ApacheFlink to power their real-time analytics and data quality. By building a SQL-based self-service platform, they democratize stream processing, enabling everyone from data scientists to business analysts to create real-time pipelines effortlessly. Discover how Grab overcame the challenges of slow reporting and ensured data quality with a proactive monitoring system. Dive into the details and learn how you can transform your data streams into strategic assets. Read the full story here: https://lnkd.in/g6sJ9Trx #FlinkForward #ApacheFlink #DataQuality #RealTimeAnalytics #BigData

⚙️ Flink Agents: Smarter Cloud Ops, Coming Soon! Cloud operations rely on rule-based systems—but what if AI could diagnose complex issues like garbage collection pauses or network glitches autonomously? Flink Agents (in development) will: 🧠 Use RAG to query operational knowledge bases 🔍 Automate diagnostics (e.g., log searches, node health checks) 🛠 Propose solutions for low-impact actions + request human approval for high-impact ones This is part of the roadmap for Flink Agents to transform AI-driven cloud operations. Join the journey as we build the next-gen ops platform! https://lnkd.in/g49SWRYa #AIOps #CloudInnovation #ApacheFlink #EventDriven #FutureTech

Live streams generate thousands of comments per second—traditionally requiring manual moderation. But what if AI could handle this in real time? Flink Agents (in development) will revolutionize this with: ✅ Real-time comment summarization and FAQ detection ✅ Multimodal analysis for audience demographics + sentiment ✅ Actionable recommendations: Suggest product adjustments or background music based on insights This is just one of the planned use cases for Flink Agents. Stay tuned to learn how this event-driven AI framework will redefine live streaming workflows! https://lnkd.in/g49SWRYa #FlinkForward #ApacheFlink #AI #EventDriven #RealTimeProcessing

🎉 #VLDB2025 has published the paper “Disaggregated State Management in Apache Flink® 2.0”! 🎉 🤝 Who made it happen A true community effort—co-authored by the #ApacheFlink community, Alibaba Cloud, Boston University, and KTH Royal Institute of Technology. The work decouples state and compute, slashing snapshot cost and speeding recovery—a major leap toward cloud-native, high-scale stream processing. 📜 Why it matters • Exactly 10 years after Flink’s first VLDB paper defined consistent streaming state, this new work charts the next leap: disaggregated, cloud-native state. • Shows continued academic trust in Flink and Alibaba’s decade-long contribution to open innovation. • Kicks off a new chapter—Generic Incremental Compute: ForSt + batch push-down promise lower latency AND lower cost, bringing near-real-time analytics to everyone. 🗓️ VLDB 2025 is LIVE NOW in London! Join us at the 51st International Conference on Very Large Databases (VLDB) as we explore the future of distributed systems. 📅 When: September 1–5, 2025 📍 Where: London, UK Don’t miss Industry Session 1 on Tuesday, September 2nd (9/2)—Disaggregated State Management in Apache Flink® 2.0. This session is a must-attend for data engineers building next-gen architectures! 🔗 Learn More: Download the Paper: https://lnkd.in/gqcd9BYW Read the Technical Deep Dive: https://lnkd.in/gnwB48kn Watch the full recording on Youtube: https://lnkd.in/gV-n45sq #VLDB2025 #ApacheFlink #DataEngineering #StreamingSystems #DistributedComputing Yuan Mei

Flink Forward Asia City Tour | Shanghai showed up! On Aug 16, 2025, hundreds of developers gathered for the Flink Forward Asia City Tour—an afternoon of deep tech, hallway chats, and community energy. Feng WANG opened with reflections on the development of the overall Apache Flink community and ecosystem. Xintong Song mapped Flink’s evolution in the AI era, highlighting the community's strategic focus on AI integration—ranging from real-time RAG pipelines to Flink Agents. Jingsong Lee unveiled Paimon REST Catalog, simplifying lakehouse metadata and boosting usability. Ele.me (food delivery service) team shared Flink+Paimon production lessons. Topsports China team (sports retailer) detailed the move from Lambda to a unified Flink+Paimon stack. The Taobao team showed how Apache Fluss (Incubating) powers a trillion-event/day real-time warehouse while tackling heavy-state joins and Kafka full-column costs. We closed with a hands-on lab led by Aliyun.com's expert: enterprise-grade CDC, plus a hands-on lab of Paimon+Flink+StarRocks. Thanks to our speakers, inquisitive minds who asked great questions, and story-sharers. This is just the beginning—mark your calendars for the next city tour!