vLLM

🇸🇬 vLLM Singapore Meetup — Highlights Thanks to everyone who joined! Check out the slides by vLLM’s DarkLight1337 with tjtanaa / Embedded LLM * V1 is here: faster startup, stronger CI & perf checks. * Scaling MoE: clear Expert Parallelism (EP) setup for single/multi-node + elastic EP to match traffic. * Disaggregated serving: split prefill vs. decode to tune TTFT (time-to-first-token) vs. throughput. * MLLM speedups: reuse embeddings with a processor cache, optional GPU-side processors, and encoder DP-across-TP (replicate small encoders per TP rank; shard the decoder) to cut comms overhead. Also: WEKA — vLLM + LMCache Lab + SSD for high-perf KV cache. @ASTARsg MERaLiON — deploying AudioLLM with vLLM + Ray for autoscaling & load balancing. Slides Folder: https://lnkd.in/gwVdv6-k

Hi folks - If you're in the Austin Area on Wednesday September 17th, we (PyTorch ATX) are hosting a joint meetup with the vLLM community at the Capitol Factory and we'd love to have you join us. The sessions are listed below. You'll get a solid grounding in vLLM and also learn about two really cool new ground breaking projects in the semantic router and llm-d. We have 200 people already signed up, but still have a few spots open, please help us share the event. It's going to be awesome! - https://lnkd.in/gPwt-ZQn - Getting started with inference using vLLM - Steve Watt, PyTorch ambassador - An intermediate guide to inference using vLLM - PagedAttention, Quantization, Speculative Decoding, Continuous Batching and more - Luka Govedič, vLLM core committer - vLLM Semantic Router - Intelligent Auto Reasoning Router for Efficient LLM Inference on Mixture-of-Models - Huamin Chen, vLLM Semantic Router project creator - Combining Kubernetes and vLLM to deliver scalable, distributed inference with llm-d - Greg Pereira, llm-d maintainer

🚀Join us for the Boston vLLM Meetup on September 18! Our first Boston meetup back in March was fully packed, so register early! Hosted by Red Hat and Venture Guides, this event brings together vLLM users, developers, maintainers, and engineers to explore the latest in vLLM and optimized inference. Expect deep technical talks, live demos, and plenty of time to connect with the community. 📍Location: Venture Guides office by TD Garden/North Station 🕔Time: 5:00 PM – 8:30 PM Agenda highlights: * Intro to vLLM & project update * Model optimization with LLM Compressor and Speculators * Demo: vLLM + LLM Compressor in action * Distributed inference with llm-d * Q&A, discussion, and networking (with pizza 🍕 & refreshments) 👉 Register here: https://luma.com/vjfelimw Come meet the vLLM team, learn from experts, and connect with others building the future of inference.

LinkedIn not only uses vLLM at massive scale, but also actively contributes to the community, checkout their wonderful blog https://lnkd.in/gFV6zA5J

I just ran batch inference on a 30B parameter LLM across 4 GPUs with a single Python command! The secret? Modern AI infrastructure where everyone handles their specialty: 📦 UV (by Astral) handles dependencies via uv scripts 🖥️ Hugging Face Jobs handles GPU orchestration   🧠 Qwen AI team handles the model (Qwen3-30B-A3B-Instruct-2507) ⚡ vLLM handles efficient batched inference I'm very excited about using uv scripts as a nice way of packaging fairly simple but useful ML tasks in a somewhat reproducible way. This combined with Jobs opens up some nice oppertunities for making pipelines that require different types of compute. Technical deep dive and code examples: https://lnkd.in/e5BEBU95

🚨 Attention vLLM users – last call! 🚨 The Call for Proposals for our vLLM Featured Track at Ray Summit closes this Wednesday, July 30. If you're building with vLLM in production, optimizing inference, or exploring advanced use cases — we want to see it. This track is all about showcasing real-world implementations and hard-won lessons from the vLLM community. Need inspiration? Check out last year's top vLLM talks: https://lnkd.in/gmRhSbHk Submit your proposal here: https://lnkd.in/gjvKdvFF

🚀 Big big news for multimodal devs! The transformers ↔️ vLLM integration just leveled up: Vision-Language Models are now supported out of the box If the model is integrated into Transformers, you can now run it directly with vLLM — no need to rewrite or duplicate code. Just plug it in and go. Zero extra effort Performance might differ model to model (we’re working on that!), but functional support is guaranteed Curious how to serve Transformers models with vLLM? Full docs here 👉 https://lnkd.in/d-KjqbmU #multimodal #transformers #vLLM #VLM #opensource