How to run graph queries on Apache Iceberg with PuppyGraph and Ryft

Can lakehouses and open table formats handle the real time telemetry of observability? Open table formats like Apache Iceberg and Delta Lake on object storage are reshaping how we think about warehousing and analytics, but are they ready for the demands of observability? In this post, we take a look at: • 🌊 Where Iceberg, Delta, and Parquet shine for o11y • ⚙️ Where they fall short for logs, traces, and metrics • 💡 How innovations like Parquet VARIANT, clustering, and new file formats are changing the game and pointing to a bright future The answer: its a little complicated but looking promising 🤞

Applying "Compaction" algorithm in Apache Iceberg tables. Compaction is the process of rewriting many small files into fewer larger ones to reduce overhead during query planning and execution. A common approach is the "bin packing algorithm", which tries to “pack” smaller files together into optimally sized data files while avoiding overly large ones. Iceberg tracks data file sizes and statistics in its metadata (manifest), allowing background jobs to efficiently select small files and rewrite them into well-sized files without disrupting readers. This improves scan performance and reduces file listing overhead in object stores. However, tuning parameters - such as target file size, compaction parallelism, and commit frequency is key. Setting these too low or too high can either lead to constant rewrites or suboptimal file sizes, directly affecting both ingestion throughput and query performance. Here are 5 parameters that you can tune to achieve superior results. #dataengineering #softwareengineering

We just published a short walkthrough on computing order book imbalance (OBI) in QuestDB — now powered by our new N-dimensional arrays. Instead of juggling dozens of L2 columns, you can store the entire order book as a single contiguous array and slice it in SQL like a NumPy tensor. This makes OBI—and metrics like cumulative depth or price-impact curves—trivial to express directly in SQL. 👉 https://lnkd.in/eRjbhwGt #questdb #OBI #marketstructure #SQL #array #marketdata

𝐃𝐚𝐭𝐚 𝐋𝐚𝐤𝐞𝐬 𝐁𝐞𝐟𝐨𝐫𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 - I 𝑑𝑒𝑙𝑒𝑡𝑒𝑑 𝑡ℎ𝑖𝑠 𝑐𝑜𝑙𝑢𝑚𝑛, 𝑏𝑢𝑡 𝑖𝑡 𝑤𝑜𝑛'𝑡 𝑑𝑖𝑠𝑎𝑝𝑝𝑒𝑎𝑟. 𝑊𝐻𝑌!??? Data lakes had a simple idea: store everything in one place, inexpensively. Parquet files in object storage. Schema-on-read. Freedom from rigid schemas. That was the dream...or so we thought. 𝐀𝐭 𝐬𝐜𝐚𝐥𝐞, 𝐟𝐫𝐞𝐞𝐝𝐨𝐦 𝐭𝐮𝐫𝐧𝐞𝐝 𝐢𝐧𝐭𝐨 𝐜𝐡𝐚𝐨𝐬. → There’s no consistent way to track which files belong to which table. → Deletes and updates are painful and non-atomic. → Each engine processes data differently. → Quality operations don’t scale, and your “lake” becomes a swamp That’s where Apache Iceberg comes in. Iceberg is a table format that organizes your data lake files. It keeps track of every file, schema, and snapshot in a manifest. Instead of rewriting datasets, Iceberg performs atomic updates with ACID transactions and maintains consistent reads across all compatible compute engines. 𝐘𝐨𝐮 𝐠𝐞𝐭:  ✅ Reliable deletes and schema evolution  ✅ Time travel and rollbacks  ✅ Engine-agnostic interoperability  ✅ Lakehouse performance, still using open storage 𝑾𝒂𝒏𝒕 𝒕𝒐 𝒍𝒆𝒂𝒓𝒏 𝒎𝒐𝒓𝒆? 𝑭𝒐𝒍𝒍𝒐𝒘 -  Alex Merced at Dremio Danica Fine at Snowflake Lester Martin 🥑 at Starburst Kevin Liu at Microsoft Casey Karst at Fivetran Who else do you follow for Iceberg information? #dataengineering #apacheiceberg #data

Over the last 3-4 years, I’ve written numerous deep-dive blogs on Lakehouse architecture that move beyond the 101 introductions. These pieces explore the internals, design decisions, and trade-offs that shape real-world lakehouse systems. Here are some critical ones that you should give a read. ✅ Data Quality in Apache Iceberg with Write-Audit-Publish (WAP) & Branching - https://lnkd.in/dk2EGuB8 ✅ What is Apache Arrow Flight, Flight SQL & ADBC? https://lnkd.in/gxchSSpx ✅ ACID Transactions in an Open Data Lakehouse - https://lnkd.in/g6BM3Fxt ✅ Clustering Algorithms in Data Lakehouse https://lnkd.in/dvXsxNab ✅ How to Optimize Performance for Your Open Data Lakehouse? https://lnkd.in/dY9k9SUp ✅ Puffin Format in Iceberg for Advanced Statistics - https://lnkd.in/dtFJ-4Yb ✅ Deep Dive on Merge-on-Read (MoR) in Apache Hudi, Apache Iceberg & Delta Lake - https://lnkd.in/dqByRMZs ✅ Concurrency Control Algorithms in Open Data Lakehouse - https://lnkd.in/dkbWyrFi If you find them insightful, bookmark and share. Happy reading! #dataengineering #softwareengineering

Are table format wars ⚔️ over yet? Yes, but not in the way you expect. Delta Lake seems placed in the rearview mirror. Apache Hudi? The Hudi community has also moved on from format wars as it recently introduced native Iceberg "format" support. A table “format” is simply a metadata specification to define tables. It is a means to an end and if you unpack the layers, you will find what is defined as a “table format” is actually a pretty thin layer. In 2025 it’s no longer about what “table format” you should be using, and now it's a question of how are you managing your “database on the lake”. The Hudi community calls this a DLMS - “Data Lakehouse Management System” 🦾 Our long-running comparison article, first published in 2022 is now freshly updated for October 2025 including features from the latest project releases. It remains the most comprehensive resource to compare Hudi, Iceberg and Delta, but this time, with a fresh new lens. At Onehouse, our platform excels across all 3 Hudi, Iceberg and yes, even Delta. So while it doesn’t matter too much to us which one you choose, it might still matter to you. OCT 2025 REFRESHED COMPARISON - Link in comments below 👇 #apacheiceberg #apachehudi #deltalake #datalakehouse

Not all “real-time” analytic databases are built the same. Our latest engineering blog compares Apache Pinot and ClickHouse on data latency. See the results and learn why a stream-native design beats micro-batching. https://lnkd.in/guhUsDba #kafka Chinmay Soman Gnanaguru Sattanathan

 Supercharge Your Local Analytics with DuckDB! 🚀 Tired of complex setups for quick data analysis? Meet DuckDB – the game-changer for in-process OLAP. Think of it as the SQLite for analytical workloads. DuckDB is an embeddable, in-process SQL OLAP database management system. What does that mean for you? ✅ Blazing Fast: Designed for analytical queries, it leverages modern hardware to process large datasets right on your laptop. ✅ Zero Setup: No server, no complex configuration. Just pip install duckdb and you're ready to query your CSVs, Parquets, JSONs, or even directly from Pandas DataFrames! ✅ SQL Power: Full SQL support means you can use the queries you already know and love. ✅ Lightweight & Portable: Ideal for local data science workflows, prototyping, and single-machine analytics. Perfect for data professionals, analysts, and scientists who need powerful, fast, and easy-to-use analytical capabilities without the overhead of distributed systems. Give it a try! #DuckDB #DataAnalytics #SQL #Python #DataScience #OLAP #DataEngineering

Why Apache Iceberg Can Transform Your Data Lakes 🔥 Managing modern data lakes is challenging, especially when it comes to reliability, scalability, and analytical agility. Apache Iceberg is changing the game for data engineers tackling these issues. 🤜 Why did Iceberg emerge? Traditional table formats like Hive struggled to deliver on: - ACID transactions (atomic operations, rollback support) - Schema evolution (flexibly changing table structures) - Time Travel (querying historical snapshots reliably) Iceberg solves these by introducing: - Full ACID guarantees without external metadata stores - Schema and partition evolution with minimal disruption - Efficient snapshot management for consistent analytics and rollback How does it compare? - Hive tables: Manage only basic metadata. Limited support for schema changes and no native time travel. - Delta Lake: Strong transactional support, but Iceberg's open specification and multi-engine compatibility (Spark, Flink, Trino, etc.) often make it more versatile for complex, multi-cloud environments. Need to fix a broken ETL run? Iceberg lets you roll back your table to a previous state with a single command. Have you already implemented Iceberg in production? #ApacheIceberg #Iceberg #DataLake #BigData #DataEngineering #Analytics