feat(qwen4_exp): support Qwen3.8-Flash-Next

[jason-fxz]

What

Support Qwen3.8-Flash-Next (qwen4_exp), text-only.

Model structure:

• 36 GDN (gated delta net) + 12 QSA (compressed block-sparse attention) layers
• 4-stream hyper-connection residual
• PLE n-gram embedding (47.7 GiB table, kept in pinned host memory, UVA gather)
• 512 routed experts top-10 + gated shared expert; NVFP4 (RadixArk/Qwen3.8-Flash-Next-NVFP4) and official block-FP8 checkpoints

Engine support:

• CUDA graph decode (QSA selection, PLE gather and n-gram hash all capture-safe)
• hybrid radix cache: prefix reuse with GDN/PLE state snapshots at page boundaries
• MoE expert offload / hybrid CPU co-execution
• vendored vLLM/SGLang triton kernels + an original radix block top-k

How to run

# bs=1, 50k tokens reserved for KV, rest of VRAM to the expert cache
ft serve --model RadixArk/Qwen3.8-Flash-Next-NVFP4 \
    --moe-backend offload --moe-cache-auto \
    --max-running-requests 1 --kv-reserve-tokens 50000

Performance

bs=1 decode, AIME25, with the command above. Host RAM: ~111 GiB pinned (63 GiB NVFP4 expert banks + 48 GiB PLE table), 128 GB recommended.

GPU	decode
RTX 5090 (32 GB)	65 tok/s
RTX 4090 (24 GB)	36 tok/s

[Txmade25]

A100 80 GB / sm_80 result (official FP8 checkpoint)

I tested this PR at e3b6d7bdc74da8c9f88d182415ec4e8dfbb967c3 end-to-end on one A100-SXM4-80GB (sm_80), TP=1, using Qwen/Qwen3.8-Flash-Next-FP8 (not the NVFP4 checkpoint). CUDA 13.0, torch 2.11.0+cu130, FreeToken 0.1.2.

The pre-sm_89 software-E4M3 path works. The PR's sm_80 compile gate passed unchanged, the server initialized, and no Ampere source patch was needed.

Command was the documented route plus observability flags:

OMP_NUM_THREADS=16 MKL_NUM_THREADS=16 OPENBLAS_NUM_THREADS=16 NUMEXPR_NUM_THREADS=16 \
ft serve --model /workspace/models/Qwen3.8-Flash-Next-FP8 \
  --moe-backend offload --moe-cache-auto \
  --max-running-requests 1 --kv-reserve-tokens 50000

Cache auto selected 12,949 expert slots. Each measured request was greedy (temperature=0, top_k=1, top_p=1), had zero cached prompt tokens, received a same-length 95-token warmup, and then generated 511 measured tokens.

Prompt tokens	TTFT	Effective prompt/TTFT	Decode
130	4.647 s	27.98 tok/s	69.33 tok/s
700	4.673 s	149.79 tok/s	67.60 tok/s
2,701	4.707 s	573.83 tok/s	67.92 tok/s
6,500	4.754 s	1,367.25 tok/s	68.35 tok/s

Mean decode was 68.30 tok/s (67.60–69.33). That is +3.30 tok/s / +5.08% over the README's 65 tok/s RTX 5090 number. FreeToken's engine-side steady decode counter ended around 70 tok/s.

Correctness outputs

All four remained coherent and on-topic. In particular, there was no multilingual/repetitive degeneration at 2.7k or 6.5k prompt tokens. Full measured text is pasted below; every sample stops mid-answer because it hit the configured 511-token length cap.

130 prompt tokens — coherent

The foundational distinction in banking risk management lies between solvency and liquidity, two concepts that are often conflated but represent fundamentally different threats to a financial institution. Solvency refers to the long-term viability of a bank, determined by whether its assets exceed its liabilities, ensuring that the institution can meet its obligations over time. Liquidity, conversely, is a short-term measure of a bank’s ability to meet immediate cash demands, such as customer withdrawals or payment system settlements, without incurring significant losses. While a solvent bank can theoretically survive a liquidity crunch by selling assets, a liquid bank that is insolvent is ultimately doomed because its liabilities exceed its assets. This distinction is critical because a bank can fail due to a liquidity crisis even if it is technically solvent, as seen in historical runs where depositors panic and demand cash faster than the bank can convert illiquid loans into cash.

Ledger controls serve as the primary operational defense against both fraud and operational errors, ensuring the integrity of the data that informs all other risk management decisions. These controls involve rigorous reconciliation processes, segregation of duties, and automated validation checks that ensure every transaction is recorded accurately and in real-time. Without robust ledger controls, a bank cannot accurately assess its true financial position, leading to misstated capital ratios or liquidity coverage metrics. For instance, if a ledger error overstates cash reserves, the bank might believe it has sufficient liquidity to meet a sudden surge in withdrawals, only to discover a shortfall when the error is corrected. Therefore, ledger integrity is not merely an accounting requirement but a prerequisite for effective risk monitoring, as it provides the reliable data necessary for stress testing and regulatory reporting.

Capital acts as the buffer against unexpected losses, absorbing shocks that erode the value of a bank’s assets. Unlike liquidity, which is about cash flow, capital is about net worth and resilience. When a bank suffers losses from loan defaults or market downturns, these losses are first absorbed by its capital reserves. If capital is insufficient, the bank becomes insolvent. Regulatory frameworks mandate minimum capital levels to ensure that banks can withstand severe but plausible economic downturns. However, holding excessive capital can be costly, as it dilutes returns for shareholders and may restrict lending capacity. Thus, capital management involves a delicate balance between maintaining enough buffer to protect depositors and creditors while ensuring the bank remains competitive and profitable.

Liquidity facilities and deposit insurance function as external safety nets that mitigate the risk of bank runs and systemic contagion. Liquidity facilities, often provided by central banks,

700 prompt tokens — coherent

The foundation of a stable banking institution rests on the rigorous separation of duties and the integrity of its internal accounting mechanisms. By strictly segregating transaction initiation, approval, custody, reconciliation, and exception remediation, banks create a system of checks and balances that prevents fraud and operational errors from cascading into systemic issues. This structural discipline ensures that before any ledger period is closed, all deposits, loans, reserves, collateral, and settlement obligations are fully reconciled. Such meticulous attention to detail in financial reporting, supported by consistent valuation methods and complete data lineage, provides the accurate real-time visibility necessary for management to assess the bank’s true position. Without these auditable controls, the subsequent layers of risk management would be built on flawed data, rendering capital and liquidity assessments unreliable.

Capital and liquidity serve distinct but complementary roles in absorbing shocks, and understanding their difference is critical for effective risk management. Capital acts as a buffer against unexpected losses, protecting the institution’s solvency by ensuring that assets exceed liabilities even when credit or market values decline. In contrast, liquidity refers to the availability of cash or assets that can be quickly converted to cash to meet immediate obligations, such as customer withdrawals or payment system settlements. A fundamental distinction exists between liquidity risk and solvency risk: a financially sound institution with strong capital ratios can still fail if it lacks immediately available cash to meet short-term demands. Therefore, while capital absorbs the economic impact of losses, liquid assets fund the operational flow of withdrawals and obligations, requiring separate but coordinated management strategies.

To manage the inherent mismatch between long-term assets and short-term liabilities, banks employ sophisticated stress testing and scenario analysis. These tools connect macroeconomic scenarios to specific financial outcomes, including credit losses, market movements, funding pressures, and resulting capital ratios. By explicitly defining assumptions, second-order effects, and management actions, stress tests reveal vulnerabilities that standard accounting might miss, such as the impact of duration mismatches where long-lived fixed-rate assets are funded by deposits that can reprice or leave quickly. Concentration limits further mitigate risk by reducing exposure to single borrowers, industries, geographic regions, or correlated collateral sources. This proactive approach allows institutions to identify potential funding gaps and capital shortfalls before they materialize, enabling pre-emptive adjustments to the balance sheet.

When internal resources are insufficient to meet sudden liquidity demands, contingency funding plans and external facilities provide essential backstops. These plans identify usable collateral, borrowing capacity, escalation thresholds, and communication duties, ensuring a structured response to stress. In extreme cases, a lender of last

2,701 prompt tokens — coherent

The foundation of a stable banking institution rests on the rigorous application of ledger controls and the clear distinction between liquidity and solvency. Ledger controls serve as the primary mechanism for ensuring data integrity and operational accuracy, requiring a strict separation of duties where transaction initiation, approval, custody, reconciliation, and exception remediation are handled by distinct parties. This segregation prevents fraud and errors from compounding, ensuring that deposits, loans, reserves, collateral, and settlement obligations are accurately reconciled before each ledger period closes. It is crucial to understand that liquidity risk is fundamentally different from solvency risk; a bank can be solvent, meaning its assets exceed its liabilities, yet still fail if it lacks immediately available cash to meet withdrawal demands. This distinction highlights that a sound institution can face a liquidity crisis due to timing mismatches, whereas solvency issues arise from actual losses eroding the capital base.

Capital and liquid assets play complementary but distinct roles in absorbing shocks and maintaining operations. Capital acts as a buffer against unexpected losses, protecting the institution’s equity from being wiped out by credit defaults or market downturns. In contrast, liquid assets are specifically designated to fund daily withdrawals and meet payment-system obligations, ensuring that the bank can honor its immediate liabilities without resorting to fire sales. The interplay between these two elements is critical; while capital absorbs the economic loss, liquidity ensures the operational continuity required to manage that loss. Without sufficient liquid assets, a bank may be forced to sell assets at depressed prices, which can convert a liquidity problem into a solvency problem by realizing losses that erode capital.

Liquidity facilities and central clearing mechanisms provide essential infrastructure to manage systemic risk and reduce bilateral complexity. Central clearing reduces the web of counterparty risks by interposing a central entity between buyers and sellers, but this concentration of risk requires robust governance to prevent the clearinghouse itself from becoming a single point of failure. Similarly, a lender of last resort serves as a critical backstop, capable of stopping a temporary funding shortage from escalating into a system-wide bank run. However, the availability of such facilities introduces moral hazard, where banks may take excessive risks under the assumption that they will be rescued. Therefore, these facilities must be accompanied by strict supervision and risk-sensitive safeguards to ensure they are used only in genuine emergencies rather than as a subsidy for poor risk management.

Deposit insurance further stabilizes the banking system by protecting small savers and preventing panic-driven withdrawals, but it also requires careful calibration to avoid encouraging reckless behavior. While insurance shields depositors from loss,

6,500 prompt tokens — coherent

The foundation of a secure banking system rests on the rigorous application of internal ledger controls, which serve as the primary defense against operational error and fraud. These controls are designed to enforce a strict separation of duties, ensuring that the initiation of a transaction, its approval, the custody of assets, and the final reconciliation are handled by distinct individuals or departments. This segregation prevents any single actor from manipulating records to conceal losses or misappropriate funds. Furthermore, the integrity of the financial reporting framework depends on consistent valuation methodologies and complete data lineage, allowing auditors to trace every entry back to its source. Before closing each ledger period, banks must reconcile deposits, loans, reserves, collateral, and settlement obligations. This comprehensive reconciliation process ensures that the bank’s internal records accurately reflect its external obligations and asset holdings, providing a reliable baseline for all subsequent risk management activities. Without these auditable controls, the data used for higher-level risk assessments would be flawed, rendering all downstream safeguards ineffective.

Building upon accurate internal records, the bank’s capital structure acts as the primary buffer against unexpected losses, distinguishing the institution’s solvency from its day-to-day operational cash flow. Capital is specifically designated to absorb losses that exceed expected provisions, thereby protecting depositors and creditors from bearing the brunt of asset deterioration. However, capital is not a substitute for liquidity; it is a measure of net worth, not immediate cash availability. A bank may be solvent, meaning its assets exceed its liabilities, yet still fail if it cannot meet immediate payment obligations. This distinction is critical because capital absorbs losses over time, whereas liquidity must be available instantly to fund withdrawals and payment-system obligations. Therefore, while capital ensures the long-term viability of the institution by covering unexpected shocks, it does not guarantee the ability to settle daily transactions, necessitating a separate and robust liquidity management framework.

Liquidity risk management addresses the immediate need for cash, recognizing that a sound institution can still lack immediately available funds due to timing mismatches. Duration mismatches occur when long-lived, fixed-rate assets are funded by deposits that can reprice or leave quickly, creating a vulnerability during periods of market stress. To mitigate this, banks maintain liquid assets that can be quickly converted to cash without significant loss of value. However, relying solely on market prices to reveal risk can be dangerous, as forced sales during stress may amplify losses and depress collateral values, creating a downward spiral. Consequently, banks must employ scenario analysis that explicitly accounts for assumptions, second-order effects, and management actions. This analysis helps identify potential funding gaps before

Memory

Peak nvidia-smi VRAM was 74,839 MiB / 73.085 GiB. Peak engine-process-tree RSS was 167.418 GiB.

The exact CUDA-registered host banks for the official block-FP8 checkpoint are about 160.20 GiB, not ~111 GiB:

• PLE: 20,000,000 × 2,560 FP8 bytes = 47.684 GiB.
• Experts: 4,915,840 bytes/expert/layer × 512 × 48 = 112.515 GiB.

The smaller ~63 GiB expert-bank estimate applies to NVFP4, not the required official FP8 checkpoint.

Reproducible loader oversubscription issue

One non-sm_80 issue was substantial on this RunPod host. The container saw CPUs 0–127, while cgroup cpu.max was 1360000 100000 (13.6 cores), so torch defaulted to 128 intra-op workers. The expert loader then ran at ~21 MB/s with a ~96-minute ETA. A representative 1.64 MB FP8 copy_ measured:

• 128 torch threads: 0.020 GB/s
• 16 torch threads: 89.47 GB/s

With the OMP_NUM_THREADS=16 family of settings above, the 122 GB source-shard pass completed in 75 seconds at roughly 1.6–2.1 GB/s. A cgroup-aware thread clamp in the server/loader, or documenting an OMP_NUM_THREADS recommendation, would prevent this startup cliff on quota-based containers.

[andy-yang-1]

@jason-fxz Great work!

[Txmade25]

Follow-up A100 optimization result (separate NVFP4 experiment, not a replacement for the official-FP8 number above):

With RadixArk/Qwen3.8-Flash-Next-NVFP4, --nvfp4-backend triton, and cache-auto selecting all 24,576 slots on the 80 GB A100, a warm trace showed that decode still paid the LRU kernel plus an empty fused-copy kernel 48 times/token after every expert was resident: 0.584 ms/token total.

I tested a small opt-in fast path that preloads a deterministic contiguous 512-expert slice per layer, gives the existing native-NVFP4 kernel that slice with unchanged layer-local expert IDs, and bypasses LRU/copy/prefill-buffer machinery. Greedy 511-token generations were byte-for-byte identical to the unmodified path at 130, 700, 2,701, and 6,500 prompt tokens (matching SHA-256 at every length; all coherent).

Prompt	Unmodified	Fully resident fast path
130	67.63 tok/s	75.43 tok/s
700	70.66 tok/s	75.41 tok/s
2,701	71.30 tok/s	74.85 tok/s
6,500	71.69 tok/s	74.82 tok/s
mean	70.32 tok/s	75.13 tok/s

That is +6.84% mean, with the long-context steady gain at +4.4–5.0%. Peak VRAM was 79,637 MiB, so this is specifically useful when the entire expert bank genuinely fits. The prototype is environment-gated, validates full-cache/native-NVFP4/GPU/pinned-bank conditions, and disables the prefill buffers that would otherwise overwrite the first two fixed slices.

I have the tested patch and full four-length outputs saved and can submit it as a focused PR with unit coverage if this full-residency mode fits the project's direction.