How next-gen memory systems can evolve LLMs into collaborators

In their first public blog post, Thinking Machines Lab tackles the challenge of non-determinism in LLM inference. TL;DR: The reason LLMs produce different outputs for the same input (even at temp=0) isn't just floating-point math. The primary cause is a lack of "batch invariance" in the compute kernels. Key points: 1. Problem: Achieving deterministic outputs from LLMs is a known challenge, hindering reproducibility. 2. Solution: The author argues that an input's output is affected by other inputs it's batched with. The fix is to enforce batch-invariant kernels. 3. Impact: This allows for truly reproducible inference results, regardless of server load. Why it matters: This is fundamental for debugging and deploying LLMs in critical applications where consistency is required. Great insights from their debut post. Link: https://lnkd.in/gudxauWi

What we often chalk up to “expected randomness” in LLMs turns out to be partly a byproduct of infrastructure choices—batch sizes, request load, how attention kernels are implemented. The idea of batch invariance stood out. If your kernel changes reduction order when batch size or KV cache state changes, then your “same prompt” no longer means “same output.” For applications that depend on reproducibility, audit trails, or consistency (e.g. RL training, regulated domains), that’s not just academic. What do you think: for your stack, how big is the gap between “probabilistic slack acceptable” vs “exact repeatability required”? Are we shifting toward expecting determinism even in deployed systems? https://lnkd.in/gbKTt9-g #AIEngineering #Reproducibility #LLMInference

This post from Thinking Machines finally clicked for me: a big source of LLM inference non-determinism is the lack of batch invariance (same prompts, different batch sizes/orders ⇒ different outputs). https://lnkd.in/giAWQDY9 We’ve seen this repeatedly in our on-prem deployments, but we used to blame only “concurrency + floating-point” effects—a misconception I also picked up from an early thread in the official vLLM GitHub. Why it matters: In the enterprise, reproducibility isn’t nice-to-have—your core business logic may depend on LLM outputs. What helps: Treat structured outputs as first-class. I recommend use structured decoding layer like #Outlines for schema-safe generation. And remember: enabling structured/constraint decoding can change performance—expect potential throughput drops, latency increases, and different batching efficiency. Benchmark with and without constraints, across libraries and batch sizes, and explicitly test for batch invariance. #LLM #Determinism #MLOps #vLLM #StructuredDecoding #Outlines #EnterpriseAI #Batching

Ever run the same prompt through an LLM and still get different answers? Thinking Machines Lab’s first blog: Defeating Nondeterminism in LLM Inference, which explains why: the real culprit isn’t floating-point math or concurrency, but batch size. Different batch sizes change how kernels perform reductions, which leads to different outputs, even with deterministic kernels. Their solution: make kernels batch invariant so results stay consistent and reproducible. Check out their blog: https://lnkd.in/e7F2pSDv

Defeating nondeterminism in LLM inference Great read by Horace He & Thinking Machines: https://lnkd.in/g-qxAJUt TL;DR • The main culprit isn’t just “GPU concurrency + floats.” It’s that many kernels aren’t batch-invariant—results change with batch size/scheduling, so outputs vary even at temp=0. • Make RMSNorm, matmul, and attention batch-invariant; for attention, use a fixed split-size Split-KV so the reduction order doesn’t depend on load. • Demo: 1,000 greedy completions → baseline gave 80 unique outputs; with batch-invariant kernels, all 1,000 identical. Usable performance with their vLLM/FlexAttention setup and kernels. My 3 quick takeaways: 1. Determinism should be an opt-in flag in inference stacks (eval/safety/RL), not an afterthought. 2. Benchmarks must declare batch/scheduler and determinism settings—otherwise we’re comparing apples to mangoes. 3. The real win is upstreaming batch-invariant kernels (vLLM, Flash/ FlexAttention, vendor libs) so reproducibility becomes default. #LLM #MLOps #Reproducibility #Inference #vLLM #AIEngineering

A must-have tool for ML scientists and researchers is here! 🔥 torchao neatly unifies model optimization techniques into a PyTorch-native workflow. Key Features Integrated optimization pipeline Connects diverse techniques such as FP8 training, QAT (Quantization Aware Training), PTQ (Post Training Quantization), and sparsity into one seamless flow. Wide low-bit dtype support INT4, INT8, multiple FP8 variants, and MXFP families — all enabled through custom tensor subclasses, making it a true playground for low-bit experimentation. Performance & efficiency FP8 training → up to 1.5× speedup LLaMA-3 8B with autoquant (INT4 weight only) → 97% faster, 73% VRAM reduction QAT → retains 96% accuracy compared to PTQ XNNPACK integration → improved on-device generation performance https://lnkd.in/gGzPhX9d

Thinking Machines Lab recently released a great article on Defeating Nondeterminism in LLM Inference. Same prompt, new answers, again. LLMs still drift, even when you do everything right. Here are my 11 KEY TAKEAWAYS from the Blog. 1. Setting temperature to zero is not enough.    ↳ Even with temp 0, LLMs can spit out different answers for the same prompt.    ↳ This breaks trust, makes debugging a nightmare, and ruins RL training. 2. Hosted APIs and open-source servers both drift.    ↳ You can run the same model, same prompt, same settings and still get different completions.    ↳ This is not just a cloud problem. It happens everywhere. 3. Floating-point math and concurrency are not the only culprits.    ↳ Most people blame floating-point errors or server load.    ↳ But the real problem runs deeper. 4. The forward pass is usually deterministic until batch size or server load changes.    ↳ Change the batch size, or run under heavy load, and outputs start to diverge.    ↳ This is called “lack of batch invariance.” It’s the silent killer of determinism. 5. Batch invariance is the missing piece.    ↳ Outputs must not change with batch size or chunking.    ↳ If your kernels are not batch-invariant, you will never get true repeatability. 6. Common kernels like matmul and RMSNorm break batch invariance.    ↳ These are the workhorses of LLMs.    ↳ But their standard versions split up work in ways that make results drift. 7. Atomic adds are not the main problem anymore.    ↳ Modern libraries have mostly solved this.    ↳ The real issue is how reductions are split across cores. 8. The fix: keep reductions inside a core, avoid split reductions.    ↳ This means a small speed hit, or using one kernel config for all shapes (about 20% slower).    ↳ But you get trust and repeatability in return. 9. Attention needs special care.    ↳ Update KV/page tables before launching kernels.    ↳ Use a fixed split-size for Split-KV.    ↳ These tweaks make attention batch-invariant too. 10. Batch-invariant ops now exist.     ↳ With Qwen3-235B at temp 0, 1000 runs gave 80 unique completions until batch-invariant kernels, then all 1000 matched. 11. Determinism enables true on-policy RL.     ↳ Sampling and training become bitwise-identical (KL ~ 0).     ↳ This unlocks new levels of reliability for RL and debugging. WHY THIS MATTERS? Determinism means you can trust your results, debug with confidence, and build RL systems that actually work as intended. WHO’S PUSHING THIS FORWARD? Thinking Machines (Mira Murati + team) is tackling this at the kernel level and it is open, transparent, deterministic by design. Not hand-wavy “AI insights,” but concrete engineering: tracing how batch effects cascade from embeddings → softmax, then retrofitting batch-invariant matmul/normalization/attention. It’s the kind of work that actually moves the field. Checkout the Full Blog here: https://lnkd.in/dA9jaTAT

Wrote this blog with me getting started with Cryptanalysis (i.e., finding out weaknesses and strengths of the cryptosystem rather than implementing them). This work presents analysis of a "Post-Quantum" and "Fully Homomorphic" system based on the LWE Problem. Despite being secure againt Quantum Computers, it can potentially be vulnerable to Machine Learning if lazily implemented. Now, I intend to get towards Algebraic Analysis of such systems, like BKZ attack, Lattice Reduction, etc. Blog: https://lnkd.in/gwF-DXWZ GitHub repo: https://lnkd.in/gH68byAK

🚀 PyTorch/TorchRec v1.3.0 is here This release focuses on improving training efficiency and scalability for large-scale recommender systems. Key highlights include: * Faster Training: New training pipelines like Fused SDD overlap the optimizer with embedding lookups, leading to significant performance gains, especially with heavy optimizers. * Dynamic Sharding: The new Resharding API allows for on-the-fly tuning of sharding plans during training, reducing downtime and offering more flexibility. * Improved Hashing: A novel Multi-Probe Zero-Collision Hashing (MPZCH) solution addresses the common problem of hash collisions in sparse embedding lookups. This is a game-changer for large models, offering a scalable and performant solution. Also introducing support for K/V (Key Value) based ZCH to allow separate dense and sparse scale up, without being restricted by GPU resources. * Delta Tracker: The new ModelDeltaTracker helps with tracking unique IDs and their embeddings, which is a key component for online training scenarios. TorchRec continues to be a powerful tool for building high-performance recommenders. Check out the full release notes https://lnkd.in/gdy4juci to learn more! #PyTorch #TorchRec #AI #RecommenderSystems

LLM's context rot is now proved. Did you notice how sometimes LLM starts to ignore instructions or particular text chunks? I clearly noticed that when was working on automatic documenting of source codes by Qwen 2.5 coder using H100 GPU. LLM struggled to follow instructions or failed to work with sources for file sizes over 7k. Even with duplicated instructions in the end of the prompt. ⚠️ But it appears that there are the same problems for bigger scale too. Here 2 benchmarks that show how big closed and open models struggle to work with bigger context. Generally after hitting 120k window size every model starts to decay. 🔗 Fiction.liveBench: https://lnkd.in/dsartXTA 🔗 NoLiMa: https://lnkd.in/dtaT_GHF 🤔 How to fight it? If you want your intellectual system to have move accuracy, be more stable and predictable, context engineering steps into the game. The goal is to craft the given context size (usually up 128k) with high quality material. Crafted just in the right time and place. ➡️ What will help: - advanced RAG techniques like requests augmentation, hybrid search, routing, reranking; - graph-based agentic workflows; - hierarchical AI-agents when each sub-agent works within it's own context window; - memory management: history cleaning, summarization, dynamic internal state; - remembering rules or success cases. 💡 And do not forget about evaluation using DeepEval, Ragas, MLFlow or similar instruments. More on RAG advanced techniques: https://lnkd.in/diD8gwCA.