#gpu
2 papers
-
inspiration
Prefill and Decode Want Different Machines
LLM inference has two phases with opposite resource profiles. Splitting them onto separate GPUs removes their interference and lets you tune each for its own latency target.
-
inspiration
Flash Attention Is an IO Problem
Standard attention is slow not because of arithmetic — it is slow because of memory traffic. Flash Attention solves the IO problem, not the compute problem. That distinction matters for how you think about every inference optimization that follows.