feat(server): enable reasoning_effort on /v1/chat/completions - #2
Merged
Conversation
jason-fxz
changed the title
jason-fxz
force-pushed
the
feat/chat-reasoning-effort
branch
from
August 14, 2026 04:09
190747b to
4ffcc10
Compare
ezutfen
referenced
this pull request
in Zutfen-LLC/FreeToken
Aug 27, 2026
* bench: add reproducible InferSwarm Phase 0 baseline harness Tooling and instrumentation only for InferSwarm issue #2 (Phase 0 baseline). No distributed execution, and no benchmark results: this branch was developed without the target RTX 3060, so it claims no measurement. Harness (benchmarks/inferswarm_phase0/, entry point benchmarks/phase0_baseline.py) - Executes the exact B1-B5 sweep declared in the InferSwarm Phase-1 success criteria, passing every value the criteria fix explicitly: --nvfp4-backend is always stated (EngineConfig defaults it to "triton", not "auto"), and --moe-cache-auto is always stated (the CLI, not the dataclass, applies it). - CORRECTNESS_REFERENCE is a separate subcommand: fixed configuration, explicit resolved NVFP4 backend, fixed --moe-cache-size, --moe-cpu-layers 0, greedy, full output text retained. Never selected by speed, never a comparator. - Frozen workload manifest (JSON + schema) pinning per class the fixture, its sha256, output-token count, sampling, ignore_eos and chat-template settings. W1/W3/W4 fixtures come from issue #3; only a clearly-labelled smoke-test example ships here. - Precommitted protocol: 2 warmups + 10 measured generations per (arm, class), recorded execution order, distinct session ids, reversed traversal for a second session. Overrides require --dev-smoke and stamp the run NON-CANONICAL. - Provenance capture with explicit nulls-plus-reason, a refusal to start a canonical run on missing provenance, and a refusal of any model revision that is not a 40-hex commit SHA. - Raw artifacts: one JSONL line per generation (warmups tagged, never dropped) with full inter-token timings; run status derived from expected-vs-observed counts so a summary cannot hide a missing repetition. No ratio is computed and no baseline is selected by the runner. - Hardware profile subcommand (GPU identity, PCIe link gen/width, topology, ft bench bw) plus a single-expert NVFP4 decode-GEMV microbenchmark, marked diagnostic-only. Runtime instrumentation (the narrowest additions the above needs) - freetoken/engine/runtime_report.py: the engine's RESOLVED configuration, read back off the live engine rather than re-derived - resolved MoE backend, resolved NVFP4 backend and whether the flag was inert for the executing expert path, whether _auto_cpu_layers locked layers, resolved cache slots/bytes, and whether the Marlin 992-slot cap applies and whether it bound. Shipped on the readiness ack and served by a new read-only GET /v1/instrumentation. - Prefill is measured where prefill happens: CUDA events bracket the prefill model forward in Engine.forward_batch and are summed per request across chunked prefill, so prefill throughput is not prompt_tokens/TTFT. Off by default, behind FREETOKEN_INSTRUMENT_PREFILL; a multi-request prefill batch is marked shared rather than split. Tests: 156 new hardware-independent tests (subprocess/server/GPU mocked). Canonical issue: Zutfen-LLC/inferswarm#2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UPpHsotcG7hXxggzHKoYb9 * bench: backfill the resolved expert weight format into the run artifact The format is only knowable once an engine has loaded the banks, so the provenance document said "server not started yet" for the life of the artifact. Backfill it from what the arms actually reported, and record per-arm disagreement explicitly: criteria section 3 rule 4 holds the weight format constant across arms, so a mismatch invalidates the campaign rather than being smoothed into one value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UPpHsotcG7hXxggzHKoYb9 * bench: gate the Phase-0 harness on the prerequisites it precommitted to Review found that the harness could still produce an apparently canonical, COMPLETE, MEASURED artifact while a precommitted requirement had failed. The invariant it now enforces: a canonical-looking Phase-0 artifact exists only when every prerequisite, held-constant rule, workload-shape rule, instrumentation requirement, provenance requirement and repetition requirement was satisfied. Anything else is explicitly INVALID, INCOMPLETE or NON-CANONICAL. Campaign validity is now a first-class answer, separate from completeness (inferswarm_phase0/validity.py). run.json carries execution_status (COMPLETE / INCOMPLETE) alongside validity (VALID / INVALID / NON_CANONICAL) and a structured campaign_invalidations list with stable reason codes; the bare `canonical: true` boolean is gone, because it was the field a reader would mistake for a verdict. MEASURED now labels an observation, never the campaign. SUMMARY.md and run.json both lead with one of VALID CANONICAL CAMPAIGN / INVALID CANONICAL ATTEMPT / NON-CANONICAL DEVELOPER RUN / INCOMPLETE RUN, keyed off the overall campaign state rather than the repetition protocol alone -- --allow-missing-provenance leaves the protocol untouched and still makes a run non-canonical. `ft bench bw` is a session-level prerequisite, not a B2-local side effect (inferswarm_phase0/bench_bw.py). B2 resolves its fetch split from the profile AND B3's --moe-backend auto reads the same profile, so the refresh runs once before the sweep traversal in either direction -- a reversed session runs B3 first and would otherwise consume a stale profile. --no-bench-bw is refused for a canonical sweep; a failed command, an unreadable profile, or a profile benched on another card aborts before any server starts. The record pins the command, both timestamps, the return code, the GPU UUID, the resolved path, the profile contents and its sha256. Physical-GPU provenance is proven, not asserted (inferswarm_phase0/gpu.py). --gpu is required for a canonical run and resolved to a stable UUID through freetoken.gpu_select -- no second selector policy -- with both the UUID and the nvidia-smi index recorded; the resolved UUID is what every child process gets. After startup the UUID the engine reports for itself is compared with it, and a mismatch invalidates. The nvidia-smi query now asks for the index, without which a numeric selector cannot be correlated with a row. Workload shape and output length are contract, not commentary. A prompt outside its frozen class rule and a completion length that is not the requested one invalidate the canonical block; the observation is preserved and the prompt is never rewritten. The "~16,000" / "~128" tolerances the criteria leave informal are frozen in version control and reproduced in every artifact. Prefill instrumentation is a validity gate. Records are attributed by request uid (chatcmpl-<uid> against the record's own uid) rather than "newest above a sequence floor"; where the id shape is unrecognized, more than one candidate is ambiguous rather than resolved by guessing. Unavailable, disabled, missing, ambiguous, shared-batch and unusable timings each carry a stable code and each invalidates a canonical measured repetition. Warmups are exempt: they are discarded by construction. CORRECTNESS_REFERENCE forces greedy request sampling on the same frozen prompt fixture and records the override. --sampling-defaults none was not sufficient: the manifest states sampling in every request body and a request-level value beats a server default, so a realistic performance sampling would have made the correctness reference sampled -- with no seed to make it reproducible. The performance sweep keeps the manifest's frozen sampling. The single-expert microbenchmark now measures a single expert: top_k=1, timed directly. The grouped top-k step is retained as a separately named diagnostic and is never divided by top_k, because experts inside a grouped call execute concurrently. Both it and the new device-memory-bandwidth benchmark bind the process through FreeToken's own gpu_select path and record the UUID they actually bound, refusing rather than misattributing. Device (VRAM) bandwidth is measured, which issue #2 asks for and `ft bench bw` does not provide -- its ceilings are host DRAM and the PCIe link. A device-resident D2D copy over a working set far beyond L2, CUDA-event timed after warmup, every repetition reported, byte accounting stated both ways, labelled MEASURED and diagnostic-only. Provenance refusals: a dirty FreeToken checkout cannot be reproduced from its commit SHA, so a canonical run refuses it and names the modified paths; --inferswarm-commit must be a full 40-hex SHA; a Hugging Face snapshots/<sha> path and its models--<org>--<name> cache entry are reconciled with --model-revision / --model-repository and a disagreement is a refusal, while a non-snapshot path records "cannot cross-check" rather than guessing. Resolved configuration is audited against criteria section 2.3, and the report gained the fields it was missing: the resolved --max-prefill-length and --cache-type (both on SchedulerConfig, and --cache-type is rewritten by _adjust_config) and the resolved hybrid fetch fraction. A missing or null required field invalidates, as does a held-constant value differing across arms or B3 resolving to a MoE backend -- or an NVFP4 expert path -- that coincides with neither B1 nor B2. --dry-run now reports what would refuse the run, so a plan cannot read "CANONICAL" while it would abort. Tests: 169 new hardware-independent tests (294 in tests/benchmarks/, up from 125), covering every refusal and every invalidating path above. Canonical issue: Zutfen-LLC/inferswarm#2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbyUuzs7m7peuxDwfjALco * bench: enforce usable Phase-0 NVFP4 calibration * Enforce canonical Phase 0 model and GPU * test: isolate Phase-0 microbench fake torch --------- Co-authored-by: Claude <zutfen@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
scooter-lacroix
added a commit
to scooter-lacroix/FreeToken
that referenced
this pull request
Aug 29, 2026
…tch end multi-minute first-request stalls Forensics (2026-08-28/29, ridge64-67): the 'minutes to first token' class had TWO stacked causes, both paid on the first request after boot/idle: 1. Cold Triton autotune: every novel extend length (fla chunk_fwd keys on BC) and KV-depth bucket sweeps whole autotune grids INSIDE the request's forward. Measured first-hit stalls: 46s/70s/129s/270s on lengths 575/266/ 255/505; identical replays 3-6s. The existing engine warmup covered only [80,128,256,512,1024] with hand-built batches -- no +/-1 divisibility shoulders, no decode steps, no depth walk. 2. Page-cache eviction of the mmap'd expert banks (checkpoint lives on a shared spinning disk): raw-socket sentinel probe stalled 135.5s while the worker faulted 81MB (0.6MB/s) behind concurrent desktop IO. Warmup depth walks ran disk-bound at ~250 tok/s vs ~2000+ page-cache-warm; the '416.6s greeting' maestro session and the 4m23s prefill->decode gap match this contention profile. Fixes: - scheduler/warmup.py + Scheduler._prefill_warmup: 33-class extend-length ladder (1..16, 2^k +/-1, max_extend) each with one decode step, plus a chunked depth walk to serving depth through the REAL request path (admission, GDN slots, drain/free) -- run in launch.py BEFORE the ready ack so clients see loading, never an idle SSE stream. Bounded, env-gated (FREETOKEN_PREFILL_WARMUP, FREETOKEN_WARMUP_MAX_DEPTH), exception-safe (failure logs and serving continues), emits boot progress. - utils/prefetch.py: posix_fadvise(WILLNEED) the whole checkpoint at boot + KeepResident daemon re-advising every 120s (FREETOKEN_PREFETCH_MODEL, FREETOKEN_KEEP_RESIDENT_S). Steady state costs no disk IO; restores evicted pages at background priority between requests. - scripts/stall_sentinel.py: 45s raw-socket pipeline probe with per-probe api/worker read_bytes deltas; any response >10s snapshots per-thread state/wchan + io (the kernel-level evidence py-spy cannot get at ptrace_scope=1). This is what caught cause FlashML-org#2 in the act. - mtp_probe reads in _process_last_data/_free_req_resources now tolerate engine-less scheduler shells (fixes 5 pre-existing test failures). - step-probe clock fix (wrapper-owned timer; was reading _forward's _t0). Verified on ridge67 (fresh boot, warm triton cache, prefetch filling): x14 503s->12.5s (cache still filling) -> subsequent hits ~3s; x13 46s->3.0s; para 70s->2.5s; ~8.5k-token prompt 75.6s(yesterday, 1.4k!)->11.4s for 8.5k (~750 tok/s wall-clock including queue+detokenize); depth walk 280s->149s. Known issue parked: full-depth (>=~50k) warmup walk hits an HSA hardware exception -- depth capped via FREETOKEN_WARMUP_MAX_DEPTH=40960 until the >=50k-context fault is diagnosed (never exercised by any traffic before). tests: 93/93 scheduler suite (6 new warmup tests + prefetch gate test).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a typed top-level
reasoning_effortfield toChatCompletionRequestand maps it onto the per-family thinking toggle, closing the gap where/v1/chat/completionsaccepted-but-ignored the field while/v1/responses(reasoning.effort) and/v1/messages(thinking.type) already honor their protocol-native equivalents.Semantics mirror the existing Responses handling (vLLM-compatible):
reasoning_effort: "none"disables thinking viamodel_meta.think_toggle_kwargs(per-family mapping:enable_thinkingfor qwen3/glm/gemma4,thinking_modefor minimax_m3, no-op for always-on families).chat_template_kwargs.reasoning_effortfor templates that grade it (gpt-oss).chat_template_kwargsalways wins; absent effort leaves behavior unchanged.Why
Clients speaking the OpenAI reasoning dialect had no way to disable thinking on this endpoint. Concrete failure: a 64-token title-generation request burns its entire budget on reasoning and returns empty content with
finish_reason: "length".Testing
tests/server/test_openai_api.py: 3 new tests mirroring the Responses-side coverage (enable/disable/family routing/kwargs precedence);tests/server/suite green (66 passed).reasoning_effort:"none"+max_tokens:64→ noreasoning_content, real content,finish_reason:"stop"reasoning_effort:"high"→reasoning_contentpresentreasoning_effort:"none"+chat_template_kwargs:{"enable_thinking":true}→ kwargs wins, reasoning present