fabswill.com
You Asked, So I Measured: What Concurrency Actually Costs On One Local Model
Reddit pushed back on my batching post with 3 sharp, testable claims: decode is memory-bandwidth-bound, prefill batches better than decode, and prompt length changes the whole story. So I built a...