draft(qwen4): add hardware-fit runtime and modular artifact path

[nickmatteo]

Important

Upstream PR FlashML-org#232 was closed unmerged on 2026-08-29 as superseded by merged PR FlashML-org#257. This draft remains a historical/component review stack on the old PR232 base, not an upstream merge candidate. The active production runtime authority is PR257. Any upstream follow-up should be split narrowly onto current main/PR257 rather than merging or retargeting this branch wholesale.

Stack and intent

This is a draft, stacked review PR for the Qwen3.8/Qwen4-Exp hardware-fit work. Its review base remains the superseded Qwen4 implementation from FlashML-org/FreeToken#232, while the production runtime foundation is the reviewed and merged PR257 head.

The branch is a review surface, not a merge-ready monolith. Useful pieces should be split or retargeted once their upstream boundaries are agreed.

What is represented

• Qwen4 parser/router reconciliation and sparse-QSA differential oracles.
• Native NVFP4 active operators and protected BF16 splits.
• Q3_PLE_32 reader/writer with 128 logical source segments.
• Byte-exact bounded CUDA Q3 encoding, retaining the scalar codec as the reference authority.
• FTEXPERT1 sidecars and file-backed expert-source/cache integration.
• Text-only modular artifact schema and deterministic PR257 hardware-fit markers.
• Restartable Step 9B controller with immutable Git/LFS/Xet identity separation, direct-to-Z downloads, lifetime transfer accounting, strict Range resume, durable receipts, Windows atomic-sidecar recovery, B1-B5 orchestration, and isolated C6 validation.
• Production layer-1 PLE source selection requiring exactly shard_0..shard_127 plus the global scale.
• Exact active-FTW v1 terminal compatibility padding that preserves the frozen extent without changing indexed tensors or creating artifact v2.

Real Step 9B completion

Real acquisition and conversion completed under separately authorized supervisor gates:

metadata source             9 files /     57,176,714 bytes
weight source             206 files / 135,195,303,851 bytes
lifetime body ledger                 135,252,480,565 / 135,252,480,565
source retirement          false; all 215 source files retained

active FTW v1                           4,804,403,200
48 FTEXPERT1                            68,149,248,000
Q3 PLE                                  22,400,107,520
known target total                      95,353,758,720
reconciliation error                                 0

The active tensor stream naturally ended at 4,804,399,104 bytes. The frozen analytical total had counted the separately stored two-byte global PLE scale as one aligned 4-KiB page. The Stage 7K helper adds exactly one zero terminal compatibility page; it does not add a tensor or duplicate the PLE scale. All 1,698 tensor keys, offsets, lengths, dtypes, shapes, kinds, and every pre-tail byte remain unchanged.

Final artifact evidence:

• Manifest SHA-256: 06d6af4a40bbe2832fa51dc07d0c2c810c76e3c29e51f723dbe262a4cd92a876
• Manifest fingerprint: d9176faad41119608746567dc9ed9b51ef7046e128f4db921ce3c6cb792fd50d
• Active shard SHA-256: a29fd1952f72ee64d55d50b5694b121c483e092506413a53a8d7c7e7bf381426
• Q3 segments: 128
• Expert sidecars: 48

C6 static PR257 reopen

The final artifact passed isolated static validation using runtime commit 0307a6114c57b0efc61bc17688f3288fe0bf1dc7:

status                  C6_STATIC_OK
experts                 48
file-tier layers        12
resident layers         36
FileExpertSource QD      4
cuda_graph_bs           []
prefill_overlap         false
inference               false

No full model was constructed. No layer or forward pass ran. No inference, generation, serving, throughput benchmark, quality evaluation, populated-context test, retiering, or CUDA-graph optimization ran. Step 10 remains separately gated.

Validation

The isolated Stage 7K authority passed 148 checkpoint tests plus compileall and diff-check. The exact Stage 7J and Stage 7K commits replayed cleanly onto this draft branch; the resulting PR branch passed:

checkpoint suite       146 passed
compileall             PASS
git diff --check       PASS

Independent adversarial review found and closed two atomic-index cleanup issues before the Stage 7K commit: stale index-temp contamination of artifact hashing, and partial cleanup before conflicting-candidate discovery.

Review boundaries

The most useful review areas are the explicit source identity model, durable restart semantics, Q3/FTEXPERT1 component boundaries, byte-exact CUDA Q3 encoder, and the FTW terminal-padding recovery contract. Please do not merge this draft as one unit. Production runtime validation, 8K/16K behavior, throughput, quality, retiering, and graph optimization remain outside this PR's evidence.

[nickmatteo]

Important viability update: external RTX 4090 evidence at the current FlashML-org#232 head reports about 69% GPU expert-cache misses with 1,024 slots. That is not a measured SSD-tier miss rate, but it means the earlier 1%/5% FileExpertSource envelopes cannot be treated as expected behavior. For the proposed 12 tier layers, a 69% sensitivity would require 1,242 aligned record reads/s (3,444 MB/s) at 15 tok/s, above the current warm, effectively serial Stage 6 path. Real checkpoint acquisition therefore remains technically blocked pending routing-informed tier selection, another fit-valid placement, or bounded asynchronous FileExpertSource evidence. The draft body now carries the full distinction and table summary; real payload transfer remains zero bytes.

[nickmatteo]

Step 10 / Windows runtime status update (2026-08-29):

• The immutable artifact manifest remains 06d6af4a40bbe2832fa51dc07d0c2c810c76e3c29e51f723dbe262a4cd92a876.
• A mutable 18-file / 30-resident placement was validated to preserve all default file-tier layers while restoring a safe host-memory reserve.
• Stage 10B added Windows-only internal ZeroMQ loopback TCP endpoints while preserving POSIX ipc:///tmp/... byte-for-byte. Scoped runtime commit: 8e07d05b73fd3207b1ac24d35d18227c1883ccca (parent 2ffd16d6abd7e0951ae5ac5beff6408f26ae2fc7). Validation: 16 focused Windows endpoint tests, 551 server tests, and 26 placement/Stage7E tests passed.
• Real startup then succeeded: shared internal loopback map propagated to spawned workers, active FTW loaded, all 30 resident banks loaded, CUDA graphs stayed disabled, and /health reached ok with 4.09 GB beyond the mandatory physical-RAM reserve.
• The first 8K-target request reached the real scheduler but produced no token. It exposed two separate Windows blockers outside the ZMQ patch: pyzmq on the Proactor event loop (add_reader unavailable), and malformed NVCC host-compiler option encoding in the first JIT embedding/index build.

So the evidence classification is: Windows ZMQ remediation PASS; Step 10 8K/16K BLOCKED before first-token execution. No quality, throughput, TTFT, or decode claim is being made. The owned runtime tree was torn down cleanly. These Windows follow-ups should remain narrow rather than merging this historical draft stack wholesale.

[nickmatteo]

Step 10 Windows runtime update (2026-08-29, follow-up):

• Stage 10C commit e338c7ef8a8c228d6e15d7b280295c2cdf4742cb fixed the initial pyzmq policy and CUDA JIT flag issues; the exact production index kernel compiled and matched Torch.
• Real first-forward testing exposed two narrower seams, fixed in Stage 10D commit b52343e75a16f07b2f2ee1e348c50c508e7125e4: Uvicorn's explicit Windows Proactor loop factory now receives a Selector factory, and fast_index_copy_multi_1024_8 now compiles under VS 2022/CUDA 13.1 and matches a two-bank Torch reference. Regression evidence: 31 focused + 592 broader tests passed.
• A real exact 8,192-token context then completed successfully: 8,191 prompt tokens + 1 generated token ( test), measured TTFT 105.969 s and prefill 66.38 tok/s. CUDA graphs remained disabled; physical RAM stayed at least 1.57 GB above the mandatory reserve during the run.
• The required matched repeat is still blocked before forward: the first radix-prefix-cache lookup lazily invokes a CPU-only AOT build that bypasses the CUDA wrapper, selects VS 2026, and sends GCC -std=c++20 -O3 flags to MSVC. It emitted zero tokens and is classified VOID.

Current classification: single real 8K run MEASURED PASS; deterministic 8K repeat BLOCKED; 16K NOT STARTED. No quality or 16K claim is being made. The owned model process was torn down and RAM/GPU returned. The next patch should remain narrowly scoped to the Windows CPU-AOT toolchain path rather than merging this draft stack wholesale.

[nickmatteo]

Step 10 Windows validation update (measured real artifact evidence):

• Final local validation runtime: e957084ceebd57581662eab01302f8dc8faf7ca1 (not pushed to this draft in this update).
• Artifact manifest: 06d6af4a40bbe2832fa51dc07d0c2c810c76e3c29e51f723dbe262a4cd92a876.
• The original 30-resident placement breached the 6 GiB physical-RAM reserve during startup and was stopped before any request. The mutable placement was safely adjusted to 28 file-tier / 20 resident layers (QD4), without changing artifact components.
• Exact 8K repeat: 8,191 prompt + 1 output, HTTP 200, finite test, response SHA-256 88c40a73942d0e0bdf8b557d3d01bcebf40627bd55c567a0ce6973d0f9e08cc3, matching the earlier successful 8K run.
• Exact 16K validation: two cold server launches, each 16,383 prompt + 1 output, zero cached prompt tokens, HTTP 200, byte-identical test, same response SHA-256 above.
• 16K TTFT/elapsed: 301.687 s and 278.484 s; measured prefill 50.32 and 55.38 tok/s. These are validation observations, not a general throughput claim.
• Lowest sampled physical availability across 16K was 16,863,305,728 bytes, safely above the 6,442,450,944-byte reserve. CUDA graphs remained disabled and prefill overlap remained false.
• No stock FP8 PLE, PinnedUVA, network, or full-model fallback was observed. The expected Windows pure-Torch router fallback remained active because triton_kernels has no Windows wheel.
• All owned validation processes were torn down; port 1927 is free. No 64K, quality, concurrency, or optimization claim is being made.

Durable local evidence: docs/FREETOKEN_STEP10_PR257_REAL_8K_16K_VALIDATION.md and results/FREETOKEN-QWEN4-001/step10_pr257_real_8k_16k_validation.json.

[nickmatteo]

Stage 12A low-context router update (RTX 5070, frozen 28-file/20-resident QD4 placement, graphs off):

• The vendored Triton BF16 top-10 router is 2.23x faster at 511 rows and 7.63x at 8191 rows in isolation.
• Across 49,152 production-shaped BF16 rows, Torch and Triton selected the same expert set on every row; weight-by-expert max absolute error was about 9e-8. PyTorch's positional order for exact ties is unspecified, while the Triton helper canonicalizes ties by expert ID.
• Matched real-model 512/1024/2048/4096-token runs produced identical response bodies and SHA-256 hashes, but throughput deltas were only -0.19%, -0.17%, +0.76%, and +0.24%.

Conclusion: the router patch is not being promoted; the real delta is noise. The dominant cost is eager full-layer expert materialization (~68.15 GB moved per prefill across 48 layers), which makes 1K-4K wall time nearly flat. The next optimization lane should instrument FileExpertSource SSD reads, resident HostBank transfers, and GPU-cache activity separately, then reduce that fixed materialization cost. No 8K confirmation was run because the lower-context gain gate did not pass.

[nickmatteo]

Stage 12C routed-expert prefill update (real artifact, RTX 5070, frozen 28-file/20-resident QD4 placement, graphs and prefill overlap off):

• Local candidate commit: b82037d2f358133d9337459247fe8e83af624179 (not pushed to this draft in this update).
• Stage 12B telemetry proved the eager prefill path materialized all 512 experts at every layer: 68.14 GB moved for a 511-token request while routing only 91.23 unique experts/layer on average.
• The opt-in --moe-prefill-routed-set path deterministically compacts exactly the routed experts into cache slots and executes the unchanged production expert kernels. In the diagnostic it reduced bytes moved to 22.90 GB (-66.39%) and file-tier reads to 9.27 GB (-76.68%), with identical response text.
• Matched telemetry-off lower-context results: 512 tokens 95.88 -> 66.06 s (+31.10%); 1K 53.14 -> 17.26 s (+67.52%); 2K 55.23 -> 18.18 s (+67.07%); 4K 59.08 -> 27.44 s (+53.56%). Response text matched at all four points.
• Cold 8K: 143.66 -> 111.88 s (+22.12%), 8,191 uncached prompt tokens, exact test response.
• Cold 16K: 274.02 s for 16,383 uncached prompt tokens. This is only +1.60% versus the best accepted 278.48 s repeat (+5.54% versus the two-run mean), so the materialization optimization is effectively saturated by 16K.
• Physical RAM remained above the 6 GiB reserve; CUDA graphs stayed disabled; the owned server was torn down and port 1927/GPU memory were released.

Conclusion: retain routed-set materialization as an opt-in candidate, with the measured sweet spot at 1K-4K and a useful 8K gain. For 16K, the next justified lane is the exact Windows router/attention path rather than further SSD queue-depth tuning. The runtime still reports the known pure-Torch router fallback because triton_kernels has no Windows wheel.