How to create event-driven Apache Airflow DAGs with Kafka

Why Spark jobs and zero-copy Kafka won't cut it for building Iceberg tables from real-time data. • High latency: hours or days between data landing in Kafka and Iceberg table updates. • Complexity: writing finicky code for transformations, schema migrations, and Spark management. • Small file problem: frequent writes lead to slow queries and expensive storage. Takeaway: Traditional approaches to building Iceberg tables from Kafka data are plagued by latency, complexity, and scalability issues. #icebergdatabase #kafkaintegration #datalakeengineering https://lnkd.in/grAfKQNv

Data Engineers: It's time to grab the Apache Airflow 3.0 certificates. At this moment 𝟳𝟳𝗸 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀 𝘂𝘀𝗲 𝗔𝗽𝗮𝗰𝗵𝗲 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 to manage their ETL and MLOPS workflows.  𝟰𝟳.𝟵% 𝗼𝗳 𝘂𝘀𝗲𝗿𝘀 reported that it powers 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀-𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀 within their organization, and this number is expected to grow. Getting certified in Apache Airflow puts you on the radar of these companies, opening you up for better career opportunities or bigger clients. And passing the exam isn't hard since Marc Lamberti has already prepared a ton of free modules that dive deep into Airflow 3.0 topics. If you're ready to start, check out my guide on the certification program 👇 #DataEngineering #ApacheAirflow #Astronomer #ETL

🔥 After getting a lot of ❤️ and encouragement on Part 1: Kafka Fundamentals, I’m excited to share Part 2: Kafka Connect in Action. 🔹 What is Kafka Connect? It’s a framework to move data seamlessly in & out of Kafka — no custom producer/consumer code needed. Source Connectors → bring data into Kafka (from files, DBs, APIs). Sink Connectors → push data out of Kafka (to files, DBs, dashboards). 🔹 Hands-On Demo We built a simple pipeline: 📄 alert-source.txt ➡️ 📦 Kafka Topic ➡️ 📄 alert-sink.txt Now any line added to the source file flows through Kafka and lands in the sink file automatically ⚡. 👉 In the next part, I’ll dive into Dead Letter Queues (DLQs) and how they make pipelines more resilient. ✍️ Read the full blog here: https://lnkd.in/gZ8PJFmD 🙏 Special thanks to Viktor Gamov 💖 for his invaluable insights in Kafka in Action, which inspired me to dive deeper into this topic. #Kafka #KafkaConnect #DataEngineering #StreamingData #RealTime #DistributedSystems #BigData #EventStreaming #DataIntegration #CloudComputing #ScalableSystems #DataPipelines

Why Kafka + Iceberg Will Define the Next Decade of Data Infrastructure Most teams treat Kafka and Iceberg as separate worlds - one for streaming, one for analytics. In this webinar, we’ll show how Streambased unifies both into a single queryable layer, without ETL, duplication, or lag. Join Tom Scott and Roman Kolesnev as they unpack what “zero-copy” really means and how it reshapes the Kafka Iceberg stack. They’ll cover: ✅ Why Kafka + Iceberg are defining the next decade of data infrastructure ✅ What makes “zero-copy” different from connector-based ETL ✅ How to unify streaming and historical data in one queryable layer ✅ Live demo: querying Kafka data instantly with Iceberg semantics   Thursday, Nov 6 • 1pm GMT / 2pm CET   Register here 👉 https://luma.com/l7k1o2gz Can’t join live? Register anyway - we’ll send the recording! #Kafka #Iceberg #DataStreaming #DataInfrastructure #RealTimeAnalytics #Streambased

Excited to share this with the community and hear your thoughts! 🙌 📘 Apache Kafka - Complete Deep Dive In this post, I break down Kafka’s core concepts — topics, producers, consumers, brokers, partitions, and real-world use cases — in a simple, structured way for anyone looking to understand how Kafka powers real-time data pipelines and event-driven architectures. https://lnkd.in/g2GrjgBR #ApacheKafka #BigData #DataEngineering #EventDrivenArchitecture #LearningJourney #RealTimeStreaming #TechBlog #Kafka

After weeks of trial, errors, and late nights debugging containers, I finally built my own Modern Data Lakehouse ELT pipeline completely open source. Using Apache Airflow, MinIO, Apache Iceberg, Project Nessie, Dremio, and Docker, this setup handles everything from data ingestion and transformation to versioned storage and analytics all in one unified architecture. It wasn’t smooth I ran into broken services, catalog mismatches, and endless configuration issues. But each hurdle helped me understand how the modern data ecosystem truly fits together. Though I’m not saying it’s perfect, I’m still a bit unsure about the design flow. Those who are experts in this domain I’d really appreciate your suggestions and feedback. Check out the project here 👉 https://lnkd.in/g6VsUtqW

Excited to share my latest Data Engineering Project! I built a real-time data pipeline that processes live weather data at scale 🌦️. Here’s how it works: - Pulling data from the OpenWeather API (handling ~3000 API calls/minute). - Ingesting data into Kafka Producer. - A Flink Consumer processes and aggregates the weather data every minute, then pushes it into another Kafka topic. - Another consumer stores the processed data into Postgres (via Supabase). - Finally, I integrated Grafana with Postgres to visualize real-time insights, refreshing every 5 minutes. - To orchestrate this workflow, I’m using Apache Airflow to schedule the Flink jobs at 5-minute intervals. This project helped me explore: - Building scalable streaming pipelines - Real-time data aggregation - Orchestration with Airflow - Visualization of live data Tech stack highlights: Kafka Flink Airflow Postgres Supabase Grafana 🔗 GitHub Repository I’ve open-sourced the entire project here: https://lnkd.in/g2cgcAmz Would love to hear feedback and ideas from the community to make this even more production-ready 🙌 #DataEngineering #ApacheKafka #ApacheFlink #ApacheAirflow #RealTimeData #DataPipeline #Postgres #Grafana

Wide & Narrow Dependencies in Apache Spark 👨💻 When we process big data using Apache Spark, transformations play a major role. There are two types of dependencies that decide performance: 🔹 Narrow Transformation Each partition depends on only one parent partition. No data movement between partitions. Faster execution (example: map, filter). 🔹 Wide Transformation Data from multiple partitions is required to create the next stage. Causes shuffle (data moves across the cluster). Slower and more resource heavy (example: groupByKey, reduceByKey). ✅ Key takeaway: Use narrow transformations wherever possible. Wide transformations are powerful but should be used carefully to avoid performance bottlenecks. Understanding this helps us design faster, scalable, and optimized data pipelines in Spark. #ApacheSpark #BigData #DataEngineering #ETL #DataPipelines #SparkOptimization #DataProcessing #LearningSpark

Mastering Apache Kafka – From Basics to Performance Optimization! If you’ve ever worked with real-time data, event-driven systems, or streaming pipelines, you’ve probably heard of Apache Kafka. I’ve compiled a complete beginner-to-advanced guide with concepts, examples, and performance tuning tips to help you become Kafka-ready: 🔹 Kafka Basics – Topics, Partitions, Replication, Brokers, Leaders & Consumer Groups 🔹 Example Use Cases – Website tracking, real-time stream processing, log aggregation, event sourcing 🔹 Producers & Consumers – Ack values, batching, compression & client libraries 🔹 Performance Optimization – Tuning brokers, balancing partitions, ISR (In-Sync Replicas), retention policies 🔹 Kafka Architecture Deep Dive – Logs, offsets, Zoo Keeper, producer/consumer APIs 🔹 Best Practices – Partition distribution, avoiding hardcoding, scaling strategies, server concepts 💡 Whether you’re just starting with Kafka or looking to optimize production systems, this guide gives you a clear roadmap from basics ➝ advanced performance tuning. 👉 Check it out for complete notes & hands-on practices 😁 🧐 👍 : https://lnkd.in/gyjskYZN #ApacheKafka #Kafka #EventStreaming #BigData #DataEngineering #RealTimeData #LearningCommunity #HelpingHands #AnshLibrary

Hey Snowflaker's, I have tried implementing a containerized log-based CDC (Change Data Capture) pipeline using 𝐏𝐨𝐬𝐭𝐠𝐫𝐞𝐒𝐐𝐋 → 𝐃𝐞𝐛𝐞𝐳𝐢𝐮𝐦 → 𝐊𝐚𝐟𝐤𝐚 → 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 all fully orchestrated with Docker Compose 🐳. This setup captures real-time data changes from Postgres 𝐖𝐀𝐋 𝐥𝐨𝐠𝐬, streams them through Kafka, and loads them into Snowflake, where a dynamic SCD Type 2 table keeps historical versions seamlessly in sync. 💡 Highlights: 👉 WAL-based CDC using Debezium 👉 Fully containerized with Docker Compose 👉 Kafka + Snowflake connectors integration 👉 Dynamic SCD2 implementation in Snowflake 👉 End-to-end setup scripts for quick replication If you’re exploring real-time data pipelines, CDC patterns, or Snowflake ingestion via Kafka, this repo walks through every step — from Postgres setup to dynamic tables in Snowflake. 👉 Check out the full implementation, detailed setup guide, and debugging steps here: https://lnkd.in/ggN_vmbD . . . . . . Sachin Mittaal Vishal Kaushal Rahul Jain 🇮🇳 Sebastian Flak Xebia #snowflake #debezium #kafka #snowflakekafkaconnector #logbasedcdc #dataengineer