perf(fp8): run per-tensor fp8 as W8A8 via scaled_mm

[jason-fxz]

Problem

fp8_pertensor_linear sends every M>1 to _gemm, a prefill kernel with no split-K. At decode
batch sizes its grid collapses to cdiv(N,128) CTAs, so runtime tracks K and ignores N and M —
594 GB/s flat from M=2..16, against 1334 GB/s for the M=1 GEMV and a 1461 GB/s copy roof.
Qwen3.8-27B-NVFP4 puts 7.21 of its 17.56 GB of weights on that path.

Fix

Route to torch._scaled_mm (W8A8) using the checkpoint's input_scale, which the loader
previously dropped. The scheme is chosen by (has input_scale, sm_89+) — a deployment property,
never by M, so a reply reproduces at bs=1 regardless of concurrency. Merged projections use
row-wise scaling to stay exact.

All per-tensor-FP8 projections in one forward (7.21 GB of weights), CUDA-graph captured, L2-cold:

M	W8A16 triton	W8A8 scaled_mm
1	5.50 ms	5.69 ms	0.97x
2	10.09	5.76	1.75x
4	10.08	5.77	1.74x
8	10.14	5.77	1.76x
16	10.09	5.76	1.75x
32	10.15	5.77	1.76x
64	11.19	5.75	1.95x
128	12.01	6.48	1.85x
256	15.01	8.24	1.82x
512	27.08	11.60	2.34x
1024	50.99	21.76	2.34x
2048	97.71	42.24	2.31x
4096	195.12	85.55	2.28x

End to end on Qwen3.8-27B-NVFP4: decode tps at 16 concurrent 657 → 799 (+21.7%), bs=1 −1.9%.
Prefill gains more, which shows up as TTFT −45% at bs=1, −14% at bs=16.

Caveats

• Activations go bf16 → fp8 where the new path runs (2.7e-2 rel. vs exact bf16) — the contract
  the checkpoint was calibrated for. Accuracy benchmark not run (tests/e2e/test_aime.py,
  no local AIME jsonl); smoke-checked on 8 concurrent prompts.
• sm<89 unchanged, still has the grid collapse; follow-up should add split-K to _gemm.
• New tests/kernels/test_fp8_pertensor_linear.py (33 cases) — the module had none.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.