feat(bench): ft bench decode — end-to-end tokens/s, with A/B across engine flags

[gdevenyi]

Why

ft bench bw measures one kernel in isolation over a small synthetic bank. That is the right shape for calibrating a bandwidth ratio. It is the wrong shape for answering whether a configuration is faster, because a serving step is the kernel plus the PCIe gather it contends with, plus the KV traffic, the GPU-to-CPU handshake and the scheduler, on the real model at the real cache size.

The two can disagree completely. Confining the CPU MoE pool to one NUMA node measured +42% for bf16 and +30% for ds_fp4 on the microbenchmark, and -6.7% on tokens per second serving DeepSeek-V4-Flash. Under --moe-backend cpu it measured -28%. I withdrew that change on the strength of the second number.

Answering that question took a throwaway shell script: start a server, poll until ready, curl some completions, grep the throughput out of the log, kill it, repeat. I wrote that script three times in one afternoon. That is the argument for this command.

Usage

ft bench decode --model DIR
ft bench decode --model DIR --compare moe-backend=hybrid,offload
ft bench decode --model DIR --compare moe-cache-rate=0.1,0.25,0.5 --cycles 3

  cycle 1  moe-cache-rate=0.10            39.97 tok/s  (49s)
  cycle 1  moe-cache-rate=0.25            69.51 tok/s  (44s)
  cycle 2  moe-cache-rate=0.10            40.15 tok/s  (46s)
  cycle 2  moe-cache-rate=0.25            69.45 tok/s  (41s)

How it measures

Each measurement reloads the model in a fresh process, so no variant inherits another's cache state.

Prefill is subtracted rather than ignored. The command times two generations of different lengths and takes (n-1) / (t_n - t_1), which cancels the fixed prefill cost.

Variants run in alternating cycles rather than one after another, so a machine that drifts during the run spreads that drift across both arms instead of loading it onto whichever ran second.

With one cycle the command says so and declines to call a winner:

  Only one cycle per variant: nothing separates a real difference from drift.
  Use --cycles 2 or more before believing a small gap.

--concurrency N raises the request count so batch-dependent costs become visible, and sends distinct prompts so the radix cache cannot serve the prefill.

Testing

Run on main with this PR, serving DeepSeek-V4-Flash on one RTX 6000 Ada:

  cycle 1  baseline                       15.12 tok/s  (102s)

This PR adds no unit test. The command drives a real server, so the parts worth testing are the prefill subtraction arithmetic and the cycle scheduling, both of which are pure functions and could be tested without a model. Tell me if you want that before merge.