feat(launch): support launch deepseek harness - #3
Merged
Conversation
ezutfen
referenced
this pull request
in Zutfen-LLC/FreeToken
Aug 27, 2026
* bench: add reproducible InferSwarm Phase 0 baseline harness Tooling and instrumentation only for InferSwarm issue #2 (Phase 0 baseline). No distributed execution, and no benchmark results: this branch was developed without the target RTX 3060, so it claims no measurement. Harness (benchmarks/inferswarm_phase0/, entry point benchmarks/phase0_baseline.py) - Executes the exact B1-B5 sweep declared in the InferSwarm Phase-1 success criteria, passing every value the criteria fix explicitly: --nvfp4-backend is always stated (EngineConfig defaults it to "triton", not "auto"), and --moe-cache-auto is always stated (the CLI, not the dataclass, applies it). - CORRECTNESS_REFERENCE is a separate subcommand: fixed configuration, explicit resolved NVFP4 backend, fixed --moe-cache-size, --moe-cpu-layers 0, greedy, full output text retained. Never selected by speed, never a comparator. - Frozen workload manifest (JSON + schema) pinning per class the fixture, its sha256, output-token count, sampling, ignore_eos and chat-template settings. W1/W3/W4 fixtures come from issue #3; only a clearly-labelled smoke-test example ships here. - Precommitted protocol: 2 warmups + 10 measured generations per (arm, class), recorded execution order, distinct session ids, reversed traversal for a second session. Overrides require --dev-smoke and stamp the run NON-CANONICAL. - Provenance capture with explicit nulls-plus-reason, a refusal to start a canonical run on missing provenance, and a refusal of any model revision that is not a 40-hex commit SHA. - Raw artifacts: one JSONL line per generation (warmups tagged, never dropped) with full inter-token timings; run status derived from expected-vs-observed counts so a summary cannot hide a missing repetition. No ratio is computed and no baseline is selected by the runner. - Hardware profile subcommand (GPU identity, PCIe link gen/width, topology, ft bench bw) plus a single-expert NVFP4 decode-GEMV microbenchmark, marked diagnostic-only. Runtime instrumentation (the narrowest additions the above needs) - freetoken/engine/runtime_report.py: the engine's RESOLVED configuration, read back off the live engine rather than re-derived - resolved MoE backend, resolved NVFP4 backend and whether the flag was inert for the executing expert path, whether _auto_cpu_layers locked layers, resolved cache slots/bytes, and whether the Marlin 992-slot cap applies and whether it bound. Shipped on the readiness ack and served by a new read-only GET /v1/instrumentation. - Prefill is measured where prefill happens: CUDA events bracket the prefill model forward in Engine.forward_batch and are summed per request across chunked prefill, so prefill throughput is not prompt_tokens/TTFT. Off by default, behind FREETOKEN_INSTRUMENT_PREFILL; a multi-request prefill batch is marked shared rather than split. Tests: 156 new hardware-independent tests (subprocess/server/GPU mocked). Canonical issue: Zutfen-LLC/inferswarm#2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UPpHsotcG7hXxggzHKoYb9 * bench: backfill the resolved expert weight format into the run artifact The format is only knowable once an engine has loaded the banks, so the provenance document said "server not started yet" for the life of the artifact. Backfill it from what the arms actually reported, and record per-arm disagreement explicitly: criteria section 3 rule 4 holds the weight format constant across arms, so a mismatch invalidates the campaign rather than being smoothed into one value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UPpHsotcG7hXxggzHKoYb9 * bench: gate the Phase-0 harness on the prerequisites it precommitted to Review found that the harness could still produce an apparently canonical, COMPLETE, MEASURED artifact while a precommitted requirement had failed. The invariant it now enforces: a canonical-looking Phase-0 artifact exists only when every prerequisite, held-constant rule, workload-shape rule, instrumentation requirement, provenance requirement and repetition requirement was satisfied. Anything else is explicitly INVALID, INCOMPLETE or NON-CANONICAL. Campaign validity is now a first-class answer, separate from completeness (inferswarm_phase0/validity.py). run.json carries execution_status (COMPLETE / INCOMPLETE) alongside validity (VALID / INVALID / NON_CANONICAL) and a structured campaign_invalidations list with stable reason codes; the bare `canonical: true` boolean is gone, because it was the field a reader would mistake for a verdict. MEASURED now labels an observation, never the campaign. SUMMARY.md and run.json both lead with one of VALID CANONICAL CAMPAIGN / INVALID CANONICAL ATTEMPT / NON-CANONICAL DEVELOPER RUN / INCOMPLETE RUN, keyed off the overall campaign state rather than the repetition protocol alone -- --allow-missing-provenance leaves the protocol untouched and still makes a run non-canonical. `ft bench bw` is a session-level prerequisite, not a B2-local side effect (inferswarm_phase0/bench_bw.py). B2 resolves its fetch split from the profile AND B3's --moe-backend auto reads the same profile, so the refresh runs once before the sweep traversal in either direction -- a reversed session runs B3 first and would otherwise consume a stale profile. --no-bench-bw is refused for a canonical sweep; a failed command, an unreadable profile, or a profile benched on another card aborts before any server starts. The record pins the command, both timestamps, the return code, the GPU UUID, the resolved path, the profile contents and its sha256. Physical-GPU provenance is proven, not asserted (inferswarm_phase0/gpu.py). --gpu is required for a canonical run and resolved to a stable UUID through freetoken.gpu_select -- no second selector policy -- with both the UUID and the nvidia-smi index recorded; the resolved UUID is what every child process gets. After startup the UUID the engine reports for itself is compared with it, and a mismatch invalidates. The nvidia-smi query now asks for the index, without which a numeric selector cannot be correlated with a row. Workload shape and output length are contract, not commentary. A prompt outside its frozen class rule and a completion length that is not the requested one invalidate the canonical block; the observation is preserved and the prompt is never rewritten. The "~16,000" / "~128" tolerances the criteria leave informal are frozen in version control and reproduced in every artifact. Prefill instrumentation is a validity gate. Records are attributed by request uid (chatcmpl-<uid> against the record's own uid) rather than "newest above a sequence floor"; where the id shape is unrecognized, more than one candidate is ambiguous rather than resolved by guessing. Unavailable, disabled, missing, ambiguous, shared-batch and unusable timings each carry a stable code and each invalidates a canonical measured repetition. Warmups are exempt: they are discarded by construction. CORRECTNESS_REFERENCE forces greedy request sampling on the same frozen prompt fixture and records the override. --sampling-defaults none was not sufficient: the manifest states sampling in every request body and a request-level value beats a server default, so a realistic performance sampling would have made the correctness reference sampled -- with no seed to make it reproducible. The performance sweep keeps the manifest's frozen sampling. The single-expert microbenchmark now measures a single expert: top_k=1, timed directly. The grouped top-k step is retained as a separately named diagnostic and is never divided by top_k, because experts inside a grouped call execute concurrently. Both it and the new device-memory-bandwidth benchmark bind the process through FreeToken's own gpu_select path and record the UUID they actually bound, refusing rather than misattributing. Device (VRAM) bandwidth is measured, which issue #2 asks for and `ft bench bw` does not provide -- its ceilings are host DRAM and the PCIe link. A device-resident D2D copy over a working set far beyond L2, CUDA-event timed after warmup, every repetition reported, byte accounting stated both ways, labelled MEASURED and diagnostic-only. Provenance refusals: a dirty FreeToken checkout cannot be reproduced from its commit SHA, so a canonical run refuses it and names the modified paths; --inferswarm-commit must be a full 40-hex SHA; a Hugging Face snapshots/<sha> path and its models--<org>--<name> cache entry are reconciled with --model-revision / --model-repository and a disagreement is a refusal, while a non-snapshot path records "cannot cross-check" rather than guessing. Resolved configuration is audited against criteria section 2.3, and the report gained the fields it was missing: the resolved --max-prefill-length and --cache-type (both on SchedulerConfig, and --cache-type is rewritten by _adjust_config) and the resolved hybrid fetch fraction. A missing or null required field invalidates, as does a held-constant value differing across arms or B3 resolving to a MoE backend -- or an NVFP4 expert path -- that coincides with neither B1 nor B2. --dry-run now reports what would refuse the run, so a plan cannot read "CANONICAL" while it would abort. Tests: 169 new hardware-independent tests (294 in tests/benchmarks/, up from 125), covering every refusal and every invalidating path above. Canonical issue: Zutfen-LLC/inferswarm#2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbyUuzs7m7peuxDwfjALco * bench: enforce usable Phase-0 NVFP4 calibration * Enforce canonical Phase 0 model and GPU * test: isolate Phase-0 microbench fake torch --------- Co-authored-by: Claude <zutfen@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
ft launch dsh— configure and launch DeepSeek Harness against a running FreeToken server, joining codex/claude/opencode/openclaw/hermes inPREPARERS.Design
Codex-style isolated config. The wiring lives in its own settings document instead of the user's
~/.dsh/settings.yaml, which is never read or written:~/.dsh/freetoken-launch.settings.yaml— a full dsh settings doc (llm-deepseekroute +agent-default-model), regenerated on every launch while preserving state dsh writes back (model re-picks, onboarding acks);~/.dsh/freetoken-launch.cordis.patch.yml— a one-entry--patchoverlay repointing dsh'ssettingsrow at that file for this invocation only.Launch command is
dsh --profile web --patch <overlay>(thedsh webalias rejects launcher flags); extra args pass through with alias/--profilenormalization, soft launch dsh -- --profile headless "<task>"works.llm-deepseek adapter over a custom llm-pi-ai provider. The built-in adapter replays tool-call arguments and assistant content verbatim across turns, which prefix caching depends on; the pi-ai path re-serializes arguments (parse/stringify) and nulls whitespace-only content. Non-DeepSeek model ids pass through unchanged; the
thinking/reasoning_effortfields it sends are translated server-side by the per-family mapping (effort_toggle_kwargs, #2).Context window and output cap come from the shared helpers (
_context_window,_max_output_tokens); env injectsDEEPSEEK_BASE_URL, a dummyDEEPSEEK_API_KEY(dsh rejects an empty key; FreeToken is unauthenticated), andDSH_TELEMETRY_DISABLED=1. Installer entry usesnpm install --global @deepseek-ai/dshand a Node <22 warning is printed since the published package declares noenginesgate.Testing
--dry-runagainst a stub/v1server: correct command and env plan.--config -yinto a sandboxDSH_HOMEpre-seeded with a decoy usersettings.yaml: both files written with backups, decoy byte-identical afterwards.POST /api/llm.modelsreturns the FreeToken-declared model underdeepseek-official, proving the settings repoint; the decoy settings were never loaded.Known limitation
dsh's effort picker "Off" sends
thinking: {"type": "disabled"}, which/v1/chat/completionscurrently ignores — inert for families whose template defaults thinking on. Mapping that field through the same per-family toggle is a planned follow-up.