Punit Kumar

🚀 How does micro-batch splitting reduce pipeline parallelism's idle time among multiple GPUs? When training large models across multiple GPUs, pipeline parallelism can waste a lot of compute due to “pipeline bubbles.” Here’s how micro-batching helps eliminate them 👇 🔧 The Problem — Pipeline Idle Imagine a model with 4 layers: Rank 0 handles layers 1–2 Rank 1 handles layers 3–4 Only 1 input batch ⛔ Without micro-batching: TimeGPU0GPU1t=1Compute layers 1–2💤 Idlet=2💤 IdleCompute layers 3–4 💡 Only one GPU works at a time — utilization ≈ 50%. The waiting time between GPUs = pipeline bubbles. ✅ The Solution — Micro-Batching Split the input batch into smaller micro-batches (e.g., M1–M4): TimeGPU0GPU1t=1M1 layers 1–2💤 Idlet=2M2 layers 1–2M1 layers 3–4t=3M3 layers 1–2M2 layers 3–4t=4M4 layers 1–2M3 layers 3–4t=5💤 IdleM4 layers 3–4 Now both GPUs work in parallel most of the time — GPU0 processes the next micro-batch while GPU1 finishes the previous one. 📉 Bubble Shrinkage Formula Bubble fraction=p−1m+p−1\text{Bubble fraction} = \frac{p - 1}{m + p - 1}Bubble fraction=m+p−1p−1​Stages (p)Micro-batches (m)Bubble fractionUtilization2150%50%2420%80%41615%85% ⚖️ Trade-off ✅ More micro-batches → Better overlap, higher utilization ⚠️ Too many → Small micro-batches, inefficient kernel launches & higher comm overhead Finding the sweet spot depends on hardware and model structure. 🧠 Summary ConceptExplanationPipeline bubbleIdle time when GPUs wait for dataMicro-batchingSplitting a batch into smaller chunksEffectOverlaps compute across ranksGoalKeep all GPUs busy #DeepLearning #AIInfrastructure #MachineLearning #DistributedSystems #GPUEngineering #ParallelComputing #ModelParallelism #AIOptimization #HPC #MLResearch

9

I used to think LLMs were magic. Then I started building one from scratch. Turns out, before any neural network sees your message, a surprisingly unglamorous pipeline runs first: → Split the text into pieces → Map each piece to a number → Slide a window across the sequence → Look up learned vectors → Add position signals → Hand a tensor to the transformer That's it. That's "reading" for an LLM. I wrote a full breakdown of this pipeline from basic regex tokenization all the way to positional embeddings with working Python code at every step. It also links to my BPE deep dive and a live interactive demo where you can type any text and watch it turn into tensors in real time. If you're learning NLP or building LLMs, this is the foundation everything else sits on. Read it here, along with my other ML write-ups: https://lnkd.in/eNZ7Xebu #MachineLearning #NLP #LLM #DeepLearning #Tokenization #Python #AIResearch

21

Soumith Chintala, 𝗰𝗿𝗲𝗮𝘁𝗼𝗿 of PyTorch and one of the most respected builders in 𝗺𝗼𝗱𝗲𝗿𝗻 𝗔𝗜, has joined Mira Murati ’s fast-rising startup, Thinking Machines Lab. After 11 years at Meta (Facebook), where PyTorch grew from a 𝗻𝗶𝗰𝗵𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁 𝘁𝗼 𝗽𝗼𝘄𝗲𝗿𝗶𝗻𝗴 𝗴𝗹𝗼𝗯𝗮𝗹 𝗔𝗜 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻, Chintala’s move signals a new wave in human–𝗔𝗜 𝗰𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝗼𝗻. Thinking Machines Lab isn’t just gathering top talent from Meta, OpenAI, Anthropic, and leading universities—they’re offering 𝗴𝗮𝗺𝗲-𝗰𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝘀𝗮𝗹𝗮𝗿𝗶𝗲𝘀 (up to $500K), 𝗮 𝗺𝗮𝘀𝘀𝗶𝘃𝗲 $𝟮 𝗯𝗶𝗹𝗹𝗶𝗼𝗻 𝘀𝗲𝗲𝗱 𝗿𝗼𝘂𝗻𝗱, and are aiming for a $50 billion market valuation. Their new product, Tinker, is already in use at Princeton, Stanford, and by major enterprises. Meta, meanwhile, is regrouping under the 𝗻𝗲𝘄 𝗦𝘂𝗽𝗲𝗿𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝗟𝗮𝗯𝘀 division led by Alexandr Wang, with other AI leaders reportedly on the move. Chintala’s reasoning: time to pursue “𝙨𝙤𝙢𝙚𝙩𝙝𝙞𝙣𝙜 𝙨𝙢𝙖𝙡𝙡, 𝙨𝙤𝙢𝙚𝙩𝙝𝙞𝙣𝙜 𝙣𝙚𝙬, 𝙨𝙤𝙢𝙚𝙩𝙝𝙞𝙣𝙜 𝙪𝙣𝙘𝙤𝙢𝙛𝙤𝙧𝙩𝙖𝙗𝙡𝙚.” His legacy—the PyTorch tool now thriving in classrooms from 𝗠𝗜𝗧 𝘁𝗼 𝗿𝘂𝗿𝗮𝗹 𝗜𝗻𝗱𝗶𝗮—𝗶𝘀 𝘀𝗲𝗰𝘂𝗿𝗲. #AI #TechNews #PyTorch #Meta #ThinkingMachines #Startup #AIJobs #TalentWar #Innovation #BigFunding

8

A 0 to 3 year AI/ML role is asking candidates to work with LLMs, RAG, embeddings, vector databases and Generative AI. That was the interesting part of a recent BITS Pilani, Hyderabad Campus AI/ML Engineer opening. What makes this interesting isn't the technology itself, but the experience requirement. For a field evolving this quickly, years of experience alone may not be enough to understand what a candidate can actually bring to the role. A candidate with one year of experience could already have spent that year building with the exact technologies a company needs. Someone with five years of experience could have more years on their resume, but less exposure to the problems the role actually requires them to solve. And that's where hiring gets interesting. Experience tells you how long someone has been doing something. It doesn't always tell you how well they can do what you need today. At GreenTick, this is one of the problems we think deeply about. How do you get beyond the resume and understand how a candidate actually thinks, solves problems, communicates and performs? Because when the skills required for a role evolve faster than experience can accumulate, hiring teams need better signals than years alone. The future of hiring belongs to companies that can identify capability before the resume catches up. #HiringInsights #TalentAssessment #AIRecruitment #TalentIntelligence #HiringStrategy #FutureOfWork #GreenTick

3

Wrote a blog explaining "LoRA Fine-tuning Efficiency Under Different Loss Functions." LoRA made fine-tuning cheap, but not always efficient. This post explores how loss functions like Cross-Entropy, Label Smoothing, and Focal Loss change convergence, stability, and generalization in lightweight LLMs. Read here: https://lnkd.in/gh_vm2kG

2

Excited to share that my first contribution to PyTorch has been merged into the main repository! 🔥 I contributed a fix to torch.unravel_index, a function that converts flat tensor indices into multidimensional coordinates. While looking into an edge case, I found that when the tensor shape contained a zero-sized dimension but indices was non-empty, the internal computation could perform division or modulo by zero. Instead of catching the invalid input, PyTorch would surface an unexpected RuntimeError: ZeroDivisionError. The interesting part was understanding why this case should be invalid. Internally, torch.unravel_index computes coordinates using the dimensions of shape as divisors. If even one dimension is zero, the product of the shape is zero — meaning the corresponding tensor contains zero elements. As a result, there cannot be a valid non-empty flat index into that tensor. My contribution adds an early validation check so that: → A zero-sized dimension with non-empty indices now raises a clear ValueError rather than reaching a divide-by-zero operation. → Empty indices remain valid and return empty coordinate tensors, preserving behavior consistent with NumPy. I also added regression coverage for the edge case and worked through the implementation details during PyTorch's review and CI process. One of my favorite parts of the contribution was the review discussion. A question came up about whether every zero-sized dimension should make the input invalid, which pushed me to trace the underlying arithmetic more carefully and reason through how broadcasting, coefficient products, division, and modulo interact inside _unravel_index. It's a relatively small change in a very large codebase, but the process taught me a lot about how production ML libraries handle edge cases, API semantics, testing, and backwards-compatible behavior. Really cool to see code I contributed now merged into PyTorch main. PR #191092: https://lnkd.in/gaWm6CvR Looking forward to contributing more to open source and digging deeper into the systems behind machine learning. #PyTorch #OpenSource #MachineLearning #DeepLearning #SoftwareEngineering

9

The most ironic part of all this is, the subject they deem only theoretical often define our real world. AI functioning on matrices and theorems? That's not something which you can build or realise. It's a concept, a method, a thought process. And that very thought process now impacts the whole world. Pure Science, Mechanical Engineering, Aeronautics, Food Technology, Philosophy, Psychology, all these supposed niche fields require huge brain power, intellectual capability and creativity. The applied fields and concepts are nothing without their theoretical part. We must remember that.

6

Fine-Tuning LLMs? Here’s What No One Tells You 🚨 Fine-tuning a Large Language Model isn’t just about feeding it more data. It’s like coaching a gifted student — each mistake needs its own kind of correction. In his latest breakdown, Bhavishya Pandit shares one of the most practical and clear-headed guides on LLM fine-tuning I’ve come across. He tackles the real production issues that derail models: 🧠 Overfitting – when your model memorizes instead of learning 💥 Catastrophic Forgetting – when fine-tuning wipes out general intelligence 📉 Poor Data Quality – the silent killer of performance 💸 High Compute Costs – scaling that breaks the budget 📏 Evaluation Complexity – when metrics mislead 🔁 Deployment Drift – the decay that no one plans for Each issue comes with clear symptoms, when to solve, and actionable remedies — from LoRA to rehearsal strategies to validation best practices. #LLM #MachineLearning #GenAI #AIEngineering #AI #DeepLearning #MLOps #FineTuning #AIEthics #NLP #ArtificialIntelligence #BhavishyaPandit #LinkedInLearning

81

1 Comment