Running Kimi K3 Locally Requires High-End Hardware

What does it take to run Kimi K3 locally? (it unfortunately isn’t going to run on your MacBook) Kimi K3 requires ~1.56 TB of memory for its weights alone, and the only current single-node systems that meet this bar (while supporting 4-bit precision) are NVIDIA’s B300 and AMD’s MI350X/MI355X. Unlike most earlier open-weights models, Kimi K3 is too large to run on any single Hopper or B200 node. Key takeaways when running Kimi K3 locally: ➤ It has 2.8 trillion total parameters, making it the largest open-weights model ever released ➤ Its hybrid attention mechanism reduces the amount of KV cache space required per token. For example, at 256K context on a GB300 NVL72, Kimi K3 could theoretically support over 2,000 concurrent requests using a 16-bit KV cache, compared to ~1,000 for Kimi K2.6 (when disregarding the throughput roofline) ➤ In total, running Kimi K3 locally while using its full 1M context window would require ~1.59 TB of memory for one user, growing by ~30 GB per additional concurrent user if using 16-bit KV cache ➤ The high memory requirement leads to a total system cost in the hundreds of thousands of dollars, highly dependent on specific hardware configuration and volume-based pricing ➤ The above points do not consider serving speed, which adds an additional layer of complexity - it is considerably more difficult to serve at a reasonable speed at high concurrency, vs. simply fitting the model inside available memory Realistic hardware configurations that can both fit Kimi K3’s weights and leave room for KV cache include: ➤ NVIDIA: 8x B300 or GB300; 16x B200 or GB200 (GB chips as part of an NVL72 rack) ➤ AMD: 8x MI350X or MI355X We will be publishing day-1 inference benchmarking results for Kimi K3 on a subset of capable hardware shortly, and will track how serving performance improves as inference stacks mature.