"Apache Flink 2.0 paper published at VLDB2025"

As Data Lakes grow and queries become heavier, recovery speed and reliability are more important than ever. Clumio now supports Apache Iceberg on AWS, providing durable retention and fast recovery ensuring that even the largest datasets remain accessible and ready for AI. Want to dive deeper and get details on the architecture and best practices? Reach out or just one click on the link😊 https://lnkd.in/de7JZR_V

O'Reilly + Redis report breaks down why memory is the foundation of scalable AI systems and how real-time architectures make it possible. Inside the report:  - The role of short term, long term, and persistent memory in agent performance - Frameworks and tools like LangGraph, Mem0, and Redis - Architectural patters for faster, more reliable, context-aware systems Grab it here: https://lnkd.in/ecAX3nw2

Revolutionizing AI/HPC with Open-Source pNFS: PEAK:AIO’s Bold Leap Forward! PEAK:AIO is shaking up the high-performance computing world by embracing open-source parallel NFS (pNFS) to challenge legacy systems like Lustre. As highlighted in a recent Blocks & Files article, PEAK:AIO’s scale-out, all-flash storage solution is delivering blazing-fast performance—320 GB/s from a single 2RU system, scaling linearly to superpod levels! By open-sourcing their pNFS metadata software, PEAK:AIO is fostering collaboration with industry giants like Los Alamos National Labs and Carnegie Mellon University, driving a modern, flexible file system for AI and HPC workloads. With support for CXL for ultra-low latency and plans for a unified block, file, and object protocol system, they’re building a future-proof alternative to Ceph. This isn't just an upgrade; it's a game-changer for AI training, simulations, and data-intensive workloads. Governments and enterprises are already calling for open, simple, and scalable alternatives – PEAK:AIO is answering loud and clear! Read the full article here: https://lnkd.in/g7sziert #AI #HPC #StorageSolutions #pNFS #OpenSource #DataInfrastructure #HighPerformanceComputing #CXL #Innovation #TechTrends

Week 2 of my series "System Design in Action", where I build meaningful demos to explain real-world distributed systems concepts. 🔹 Consistent Hashing Ever wondered what really happens when a server is added to a distributed system like a cache or database? I built a tiny Go demo to see exactly that, comparing Consistent Hashing against the basic Naive Modulo Hashing. It’s a hands-on look at why modern distributed systems like DynamoDB, Cassandra, and Redis Cluster all rely on hash rings. What I Built: I created a command-line application in Go that implements two hashing algorithms: - A Consistent Hashing ring with support for virtual nodes to ensure balanced distribution. - A Naive Modulo Hasher to serve as a baseline for comparison. How I Tested It: I ran a script that simulated common real-world scenarios: 📈 Scaling Up: Adding a new server to an existing 3-node cluster. 💥 Node Failure: Removing a server to simulate an outage or decommissioning. For each scenario, the tool mapped 50,000 keys and calculated exactly how many had to be moved for each hashing method. My Findings: When scaling the cluster from 3 nodes to 4: ✅ Consistent Hashing: Only ~25% of keys were remapped. The disruption was minimal and localized, affecting only the keys that needed to move to the new node. ❌ Naive Hashing: A staggering ~75% of keys were remapped, triggering a catastrophic, system-wide data shuffle. This is why consistent hashing is critical for stability. In a real system, that massive data movement from naive hashing would mean widespread cache misses, a flooded network, and a huge performance hit. Consistent hashing keeps the system calm and predictable during changes. The code is on GitHub if you want to run the demo yourself! https://lnkd.in/enEeidj5 #SystemDesign #DistributedSystems #Go #Golang #Backend #ConsistentHashing #Scalability

The Sovereign Tech Agency backs Apache Arrow’s Future with QuantStack! We are excited to share that the Sovereign Tech Fund is investing in Apache Arrow, a cornerstone of the modern data ecosystem. This funding will empower Arrow maintainers at QuantStack, Antoine Pitrou and Raul Cumplido, to advance security, modularity, and long-term sustainability. We will ensure that Arrow remains the high-performance, open foundation for data science, analytics, and scientific computing globally. https://lnkd.in/eWZ5F99b

AWS is putting Firecracker microVMs to work in two fresh stacks: AgentCore, the new base layer for AI agents, and Aurora DSQL, a serverless, PostgreSQL-compatible database it just rolled out. AgentCore gives each agent session its own microVM. More isolation, less cross-talk - solid for multistep LLM workflows packed with tool use. Aurora DSQL treats each SQL transaction like a pop-up shop. It spins up inside a snapshot-cloned, microVM-based Query Processor. That means faster starts, less memory burn, and clean-page sharing across the board. Big picture: Firecracker isn’t just for serverless anymore. It’s creeping deeper into compute and databases - fine-grained, fast, and gone when done. https://lnkd.in/e5c-xjm9 --- More like this—subscribe 👉 https://faun.dev/join

Redis — we’re everywhere. But not just in your daily life, check this out: MLOps World | Oct 8–9 | Austin Redis’ own Robert Shelton will dive into Eval-Driven Development and what it takes to build an agent from the ground up. Stop by the booth or join the talk. Redis Released | Oct 9 | London A packed lineup of innovators including Baseten, Tavily, cognee, mangoes.ai, and Entain sharing what’s next in AI and data infrastructure. SF Tech Week | Oct 10 | San Francisco A full day focused on production-ready agents — with Redis, Snowflake, Tavily, Lovable, CopilotKit, and others. Agentic AI Workshop | Oct 13 | San Francisco Hands-on sessions hosted with BeeAI and Tavily exploring how to build intelligent, connected systems.

When someone says "we use a vector database," my first question is: "Where's your system of record?" Polyglot persistence isn't a feature; it's a non-negotiable architectural choice to defeat AI hallucination. We use MongoDB for speed, RDS for truth, and VectorDBs for relevance. If you're mixing those up, your LLM is lying on a solid foundation of sand.

Consistent hashing solves a tricky problem: when scaling distributed systems, adding or removing nodes usually means rehashing almost all your data, causing massive data movement and downtime. Consistent hashing minimizes this by only moving a small fraction of keys when nodes change, making scaling smooth and efficient. The internet is full of high-level explanations of consistent hashing, but finding an actual implementation of the algorithm is rare. Finding a near-real implementation with nodes acting as cache partitions is even rarer. I built a hands-on system in Golang with a hash ring, an API to store and retrieve keys, add and remove Dockerized nodes, and real-time D3 visualization. It’s a small system, but it gives a clear view of how data moves and how nodes handle load. Check it out here: https://lnkd.in/dZvYNy5F

𝐒𝐭𝐞𝐩 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐬 𝐆𝐨𝐧𝐞 𝐖𝐢𝐥𝐝: 𝐋𝐞𝐬𝐬𝐨𝐧𝐬 𝐟𝐫𝐨𝐦 𝐚𝐧 𝐄𝐯𝐞𝐧𝐭-𝐃𝐫𝐢𝐯𝐞𝐧 𝐁𝐢𝐨𝐝𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 Today I attended an AWS session in Perth, WA where Alison Lynton shared the brutally honest story behind CSIRO’s Nature IQ biodiversity platform -and what really happens when you go “all in” on serverless. Nature IQ started as a clean, elegant, event-driven architecture but production reality had other ideas. Instead of “everything scales magically,” they hit: • Lambda throttles • Runaway concurrency • DynamoDB partition pain • AVP throttling & auth-level friction And yes - more than one Lambda-induced headache. What made the session valuable wasn’t the final diagram, but the scars that shaped it. The architecture still uses Step Functions, Lambda, DynamoDB, and S3 but now with orchestrated control, pre-computation, and multi-tenant safety baked in. Key technical insights: ✅ Job state belongs in DynamoDB ✅ Avoid env vars for runtime behaviour - use AWS AppConfig ✅ Guardrails matter more than “pretty diagrams” ✅ When upstream fails, everything fails → resilience must be intentional The big takeaway: The hard part of serverless isn’t compute - it’s control. #AWS #StepFunction #DynamoDB #AWSLambda #S3