Gadi Evron’s Post

🚨 Why Mira Murati’s Breakthrough Matters — and What It Reveals About LLMs Today’s LLMs aren’t just sometimes wrong — they’re also not reproducible. Ask the same question 5 times and get 5 different answers — even with “deterministic” settings. Murati’s new lab, Thinking Machines, just showed why: GPU kernels used in inference aren’t batch-invariant. 🧠 In plain terms: if multiple users send prompts at once, the server batches them for efficiency. Tiny differences in operation order cause floating-point shifts — which cascade into different neurons and answers. If your request runs alone, you might avoid this. But in production, they almost never do. Their fix? New kernels (matmul, attention, RMSNorm) that guarantee: same input → same output, every time. Why this matters: • Compliance: Reproducibility is mandatory in regulated sectors. • Cost: Stable outputs enable caching, cutting GPU burn. •Productization: Expect enterprise “deterministic modes.” ⚠️ But: determinism ≠ correctness. You may now get the same wrong answer every time. Reliability needs both consistency and correctness. https://lnkd.in/gYPaE2S6

Quantum Annealing with Machine Learning Tunes Penalty Parameters for Minimum Bisection Problem Solutions Researchers developed a machine learning-guided method to dynamically adjust parameters in quantum annealing solvers, substantially improving their ability to efficiently solve complex network partitioning problems and surpassing the performance of conventional algorithms. #quantum #quantumcomputing #technology https://lnkd.in/eANBGAEz

🚨𝐁𝐢𝐠 𝐧𝐞𝐰𝐬 𝐢𝐧 𝐀𝐈 𝐢𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞: 𝐌𝐢𝐫𝐚 𝐌𝐮𝐫𝐚𝐭𝐢’𝐬 𝐧𝐞𝐰 𝐜𝐨𝐦𝐩𝐚𝐧𝐲 '𝐓𝐡𝐢𝐧𝐤𝐢𝐧𝐠 𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐬' 𝐦𝐚𝐲 𝐡𝐚𝐯𝐞 𝐜𝐫𝐚𝐜𝐤𝐞𝐝 𝐭𝐡𝐞 𝐜𝐨𝐝𝐞 𝐨𝐧 𝐋𝐋𝐌 𝐧𝐨𝐧𝐝𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐢𝐬𝐦! 𝑾𝒆’𝒗𝒆 𝒂𝒍𝒍 𝒔𝒆𝒆𝒏 𝒊𝒕: 👉 Run the same prompt at temperature 0, and you still get different outputs! 👉 Traditionally, this was blamed on floating-point quirks and GPU concurrency. According to Thinking Machines (founded by former OpenAI CTO Mira Murati), the 𝘳𝘦𝘢𝘭 𝘤𝘶𝘭𝘱𝘳𝘪𝘵 is something deeper: 𝐥𝐚𝐜𝐤 𝐨𝐟 𝐛𝐚𝐭𝐜𝐡 𝐢𝐧𝐯𝐚𝐫𝐢𝐚𝐧𝐜𝐞 𝐢𝐧 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐤𝐞𝐫𝐧𝐞𝐥𝐬. Here’s what that means 👇 🔹 𝐁𝐚𝐭𝐜𝐡 𝐢𝐧𝐯𝐚𝐫𝐢𝐚𝐧𝐜𝐞 = A model should output the 𝘴𝘢𝘮𝘦 𝘳𝘦𝘴𝘶𝘭𝘵 for a prompt, no matter how requests are batched. 🔹 In reality, operations like matmul, attention, and normalization vary their internal strategies depending on batch size. 🔹 This changes reduction orders → tiny numerical differences → huge divergences in long generations. The fix? ✅ Thinking Machines built 𝐛𝐚𝐭𝐜𝐡-𝐢𝐧𝐯𝐚𝐫𝐢𝐚𝐧𝐭 𝐤𝐞𝐫𝐧𝐞𝐥𝐬 for key ops (RMSNorm, matmul, attention). ✅ Testing with Qwen-3-8B: - Default kernels → 1,000 runs = 80 unique completions. - Modified kernels → 1,000 runs = 𝘢𝘭𝘭 𝘪𝘥𝘦𝘯𝘵𝘪𝘤𝘢𝘭 𝘰𝘶𝘵𝘱𝘶𝘵𝘴. The catch: It runs slower. But for research, safety, and debugging, determinism may matter more than raw speed. As they put it: 👉 “Reproducibility is a bedrock of scientific progress.” This shift in framing—from floating-point quirks to 𝐛𝐚𝐭𝐜𝐡 𝐢𝐧𝐯𝐚𝐫𝐢𝐚𝐧𝐜𝐞—could influence how future inference engines are designed. ⚡ Question for you: Do you think 𝘥𝘦𝘵𝘦𝘳𝘮𝘪𝘯𝘪𝘴𝘮 should become a standard requirement in LLM deployment, or will speed always win? 𝐑𝐞𝐚𝐝 𝐦𝐨𝐫𝐞 𝐚𝐭: https://lnkd.in/d_uC3JgK

Handling nondeterministic LLM outputs is a real challenge—even at temperature 0. I came across 𝗗𝗲𝗳𝗲𝗮𝘁𝗶𝗻𝗴 𝗡𝗼𝗻𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝗺 𝗶𝗻 𝗟𝗟𝗠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 by Horace He and team at Thinking Machines (founded by ex-OpenAI CTO Mira Murati): https://lnkd.in/gP7bdb4h The paper starts with demoing floating-point non-associativity ((a+b)+c ≠ a+(b+c)) and then explains atomic adds. Atomic adds split tensor reductions across GPU cores and combine results into a single sum; since cores finish in nondeterministic order, the final accumulation is also nondeterministic. Paper claims that LLM forward pass is deterministic for a single query with fixed batch size, but real deployments batch concurrent requests, and varying batch sizes change reduction orders, introducing nondeterminism. The paper proposes batch-invariant strategies for RMSNorm, Matmul, and Attention to address this. 🔹 RMSNorm - The proposal is to process each element in batch on a single core 🔹 Matrix multiplication - Efficient matmul kernels rely on instructions that operate on whole “tiles,” and the internal reduction order can vary depending on which instruction is chosen. Different instructions may be used based on batch size—for instance, small batches waste compute on large tiles, and at batch size 1, tensor cores are often skipped entirely. To enforce batch invariance, one can fix a single kernel configuration for all shapes, sacrificing some performance but avoiding nondeterminism, which is generally acceptable since LLMs usually have large model dimension 🔹 Attention - Attention is more complex because it operates not only over feature dimensions but also over the sequence dimension, which encodes token positions and guides the model on where to “pay attention”. A key element here is the use of key–value (KV) caches during inference. For background, see the seminal paper “Attention Is All You Need” by Vaswani et al. The details of ensuring batch invariance in attention will exceed the length of this post, so I encourage you to explore the paper directly. Authors tests with 1000 completions (temp=0, 1000 tokens each), standard kernels produced 80 unique outputs, while batch-invariant kernels made all 1000 identical. Runtime rose from 26s to 42s, but the authors see room for further kernel optimizations I would love to see further benchmarks on longer outputs, larger KV caches, and concurrent loads Taming nondeterminism will be absolute game-changer for quality and real-world adoption of GenAI

This is one of the most important work happening in LLM reliability. Leon Chlon, PhD and his team are tackling non-determinism with mathematical rigor, not just more hardware. Their insight to fix only what's unstable rather than rebuilding everything is a powerful lesson in engineering. This open-source solution for reproducible outputs is a massive contribution to the field. See his original post for the code and details.

We solved LLM Non-Determinism pre-generation, training-free, so now you don't have to pay to replace every GPU kernel for "deterministic" ML inference. The problem: LLMs at temperature=0 aren't deterministic. When heavy machinery, medical devices, or financial systems depend on reproducible outputs, "usually consistent" isn't good enough. Thinkylabs solution: Rewrite CUDA from scratch. Wait for vendors to adopt. Hope it doesn't break with the next PyTorch update. Oh, and it still doesn't handle prompt reordering. Our solution: Accept hardware noise exists. Detect where it matters. Stabilize only those points. Zero overhead when tokens are already robust. Key innovations: - Provably deterministic when μ ≥ 2r under bounded noise - Statistical certificates (95%+ confidence) elsewhere - Selective intervention: Only pay costs at true inflection points - Observable uncertainty: Full audit trail with p_flip per token Works today: Compatible with OpenAI, Anthropic, any provider with logprobs (but built with OpenAI for now) Result: Deterministic guarantees where possible, statistical confidence everywhere else, with 1.0-1.2× overhead (vs 10-30× for naive approaches). We didn't need to rebuild CUDA. We didn't need millions in funding. We needed mathematical rigor to identify exactly where determinism is achievable and statistical methods where it isn't. Code is open source. Sometimes the best engineering isn't replacing everything - it's proving what's already stable and fixing only what isn't. code: https://lnkd.in/ep9_zYSJ readme: https://lnkd.in/eytPwmch #MachineLearning #LLM #Engineering #Reproducibility #OpenSource #SafetyCritical

How LLM Non-Determinism is Defeated Saturday night deep dive into making LLMs deterministic and my mind is blown Just spent several hours testing RABeL (Robustness-Aware Bias Elimination in Language models) - a system that provides mathematical PROOF that an LLM's output won't change. Not "probably won't change." Mathematical certainty. The results? → Same prompt = Same output, with formal guarantees → Only 1.0-1.13x API overhead for provable stability → Protection against adversarial manipulation Here's what shocked me: When I tested "Hello" as input, the system showed me the token had a flip probability of exactly 0.0. That's not marketing talk but mathematical proof that this output cannot change within defined perturbation bounds. Why does this matter for production AI? -- > Reproducible testing environments -- > Consistent responses across distributed systems -- >Formal guarantees for compliance/regulatory needs -- > No more "it usually works the same way" The trade-off is minimal computational overhead for mathematical certainty. For systems where consistency matters (IT, networking, legal, medical, financial), this changes everything. We're moving from probabilistic chaos to deterministic control in AI. And yes, I'm geeking out about this on a Saturday night because this is the kind of breakthrough that reshapes how we deploy LLMs in production. Full technical deep-dive on my blog: https://lnkd.in/dNH3jmVS #AI #MachineLearning #LLMs #ProductionAI #TechInnovation #ArtificialIntelligence #vExpertAI

An absolute must read when it comes to addressing a primary cause of non-deterministic outcomes, the isolation of this issue is almost brilliant in its simplicity. Especially when you consider all the compensating solutions and architectures that have evolved to be able to address this to meet the demands of enterprise adoption and acceptance. https://lnkd.in/gquHUKjv

In their first public blog post, Thinking Machines Lab tackles the challenge of non-determinism in LLM inference. TL;DR: The reason LLMs produce different outputs for the same input (even at temp=0) isn't just floating-point math. The primary cause is a lack of "batch invariance" in the compute kernels. Key points: 1. Problem: Achieving deterministic outputs from LLMs is a known challenge, hindering reproducibility. 2. Solution: The author argues that an input's output is affected by other inputs it's batched with. The fix is to enforce batch-invariant kernels. 3. Impact: This allows for truly reproducible inference results, regardless of server load. Why it matters: This is fundamental for debugging and deploying LLMs in critical applications where consistency is required. Great insights from their debut post. Link: https://lnkd.in/gudxauWi

The "non-determinism" of LLMs has been an issue (LLMs are notorious to produce variants of the answers to the same question) and it poses GenAI to be unsuitable in many mission-critical applications. Imagine a model providing different risk assessments for the same market conditions or varying diagnostic suggestions or treatment plans for the same patient data could lead to severe or even life-threatening consequences. Ex-CTO of OpenAI Mira Murati and her team at Thinking Machines Lab may have paved the way to this exact problem by thoroughly investigating and pinpointing the root causes of LLM inference nondeterminism. We have all been taught that the common "concurrency + floating point" hypothesis (e.g., atomic adds) is the primary reason for nondeterminism in the LLMs which may not be the case anymore. Murati's team discovered that the lack of "batch invariance" in kernels is the true culprit for this thorny issue in LLMs. Batch size refers to the number of individual user requests that are processed simultaneously by the GPUs. When the batch size changes (which happens due to varying server load from concurrent requests), each element in the batch can produce different results. By meticulously designing and implementing batch-invariant kernels for key operations (like RMSNorm, Matrix Multiplication and Attention) the team has demonstrated truly reproducible inference. This work is a significant step forward in making LLMs truly reliable and auditable for critical applications! Here's the blog post to digest at your own time: https://lnkd.in/eYHG7aUx