[Showcase & Fix] Qwen3.8-Flash-Next on Dual RTX 5090 Blackwell + Tool Call Recovery Patch #324

Hey FreeToken team! 👋

First of all, huge congratulations on FreeToken. The dynamic MoE offloading and hybrid memory architecture is phenomenal for serving 100B+ frontier models on consumer workstations.

We wanted to share our 24/7 production setup, real-world benchmarks, architecture insights, and a key bugfix for Qwen3 / Qwen3.8 tool calling under agent harnesses.

🖥️ 1. Hardware Topology & Coexistence

• Host: Proxmox VE 9.2 (IOMMU PCIe passthrough)
• VM: Ubuntu 26.04 LTS (28 vCPUs, 136 GB DDR5 RAM, 39 GB NVMe Swap)
• GPUs: Dual NVIDIA GeForce RTX 5090 Blackwell (2 × 32 GB VRAM, SM120, Driver 595.84, CUDA 13.2)
• Target Model: RadixArk/Qwen3.8-Flash-Next-NVFP4 (176.9B MoE + 51B PLE, 63.3 GB Host Expert Tables)

Multi-Engine Isolation:

• GPU 0 (32 GB): Dedicated to FreeToken serving Qwen3.8-Flash-Next-NVFP4 (29.2 GB VRAM allocated + 63.3 GB Host RAM).
• GPU 1 (32 GB): Dedicated to SGLang serving Qwen3.8-27B-NVFP4 (Dense coding model with FP8 FlashInfer KV Cache).
• Both run concurrently 24/7 with zero PCIe bus contention or host RAM exhaustion (~39 GB OS/ARC margin left).

⚡ 2. Production Service Parameters (256K Context)

ft serve \
  --model /home/chris/models/Qwen3.8-Flash-Next-NVFP4 \
  --gpu 0 \
  --host 0.0.0.0 \
  --port 1919 \
  --served-model-name RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --memory-ratio 0.90 \
  --expert-load parallel \
  --moe-cache-auto \
  --kv-reserve-tokens 262144 \
  --max-seq-len-override 262144 \
  --max-output-tokens 65536 \
  --max-prefill-length 8192 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

• KV Cache Allocation: FreeToken allocates 262,208 tokens (6.19 GiB VRAM) with moe_cache_size=3284 and num_pages=4097.
• CUDA Graphs: Successfully captures CUDA graphs for batch sizes [1, 2, 4] with 2.82 GiB free GPU headroom.
• Triton Kernels: Triton Native RoPE and sampling execute smoothly on Blackwell SM120 without FlashInfer JIT compiler stalls.

🛠️ 3. Bug Fix: Unclosed <think> Swallowing Tool Calls

When using Qwen3.8 with autonomous coding agents (DeepSeek Harness / DSH Desktop, OpenCode), the model occasionally skips emitting </think> before generating <tool_call><function=...> XML tags.

The Issue:
In freetoken/server/reasoning_parser.py, ThinkReasoningParser had no tool_start_token defined. Because of this, the streaming parser treated the entire <tool_call>...</tool_call> block as part of reasoning_content (thinking). The downstream client never received the structured OpenAI tool_calls payload and halted execution.

The Fix:
Adding tool_start_token="<tool_call>" to ThinkReasoningParser:

class ThinkReasoningParser(BaseReasoningParser):
    def __init__(
        self,
        force_reasoning: bool = False,
        stream_reasoning: bool = True,
        tool_start_token: Optional[str] = "<tool_call>",
    ) -> None:
        super().__init__(
            think_start_token=THINK_START_TOKEN,
            think_end_token=THINK_END_TOKEN,
            force_reasoning=force_reasoning,
            stream_reasoning=stream_reasoning,
            tool_start_token=tool_start_token,
        )

With this patch, reasoning is closed immediately when <tool_call> appears, and the call is emitted properly as ToolCallItem with finish_reason: "tool_calls".

👁️ 4. Multimodal Vision & Agent Performance

• Vision: 27-layer ViT visual tower loads and processes high-res screenshots via OpenAI /v1/chat/completions image_url payloads flawlessly.
• TTFT: ~3.0s – 3.5s with large system prompt tool catalogs.
• Decode Speed: ~38–54 tok/s with offloaded MoE parallel expert retrieval on DDR5 host memory.
• Stability: 100% stable in long multi-turn sessions exceeding 100K+ context.

Happy to open a PR for the reasoning parser fix if you'd like to merge it into upstream!