Skip to content

fix(ftw): preserve auxiliary expert banks - #317

Open
ErikBPF wants to merge 1 commit into
FlashML-org:mainfrom
ErikBPF:fix/ftw-auxiliary-banks
Open

fix(ftw): preserve auxiliary expert banks#317
ErikBPF wants to merge 1 commit into
FlashML-org:mainfrom
ErikBPF:fix/ftw-auxiliary-banks

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Sep 1, 2026

Copy link
Copy Markdown

Summary

Preserve expert banks that use a different quant format or row geometry from the primary bank set during FTW conversion and load. Without this, converting a heterogeneous expert layout silently writes only the primary banks.

  • add an optional auxiliary-bank contract to ExpertBanks
  • write auxiliary banks as a nested FTW checkpoint using the existing writer
  • record the nested checkpoint and model-layer mapping in parent metadata
  • validate that checkpoint and mapping metadata appear together
  • recursively load the nested banks and restore their quant format, sources, and layer IDs
  • leave existing FTW checkpoints unchanged

Provider-specific Qwen loading is intentionally separate; this PR contains only the generic FTW preservation mechanism.

Tests

  • writer test verifies per-layer nested entries and metadata
  • loader test verifies quant format, layer mapping, and exact tensor round-trip
  • Python compile, Ruff on the new test, and git diff --check pass

Full-model evidence

Matched Orion ROCm runs used 8,192-token context, 1,024 output tokens, concurrency 4, 3 rounds, and 12 requests per format:

Format Aggregate decode Aggregate E2E Median TTFT
GGUF 111.9149 tok/s 107.2566 tok/s 1.7146 s
FTW 113.6572 tok/s 108.8996 tok/s 1.6936 s

All 24 requests finished by length. Every GGUF and FTW output matched SHA-256 1ec7793ebf17b06a81148a198b7fbb5e5a68018c75982dd123cb1cf0fc2b8860. Both formats used 2,073 expert slots, 8,219 KV pages, and retained 7.71 GiB free after initialization.

The primary claim is exact conversion/load preservation. The observed +1.56% decode difference is secondary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant