[Feature Request] Support compressed-tensors per-channel FP8 checkpoints for Qwen3.5 MoE #252

Summary

I would like to request support for loading compressed-tensors FP8 checkpoints that use per-channel weight scaling, specifically for models such as:

ornith-ai/Ornith-1.5-35B-A3B-FP8

The model architecture itself appears to already be supported by FreeToken through the existing Qwen3_5MoeForConditionalGeneration / qwen3_5_moe implementation.

The missing part seems to be support for this checkpoint's FP8 serialization and quantization scheme.

Model

Hugging Face model:

ornith-ai/Ornith-1.5-35B-A3B-FP8

Architecture:

"architectures": [
  "Qwen3_5MoeForConditionalGeneration"
],
"model_type": "qwen3_5_moe"

The checkpoint uses compressed-tensors with FP8 weights:

"quant_method": "compressed-tensors",
"format": "float-quantized"

The relevant quantization scheme is approximately:

• FP8 weights
• static per-channel weight scaling
• dynamic per-token activation scaling
• selected modules excluded from quantization

This differs from the FP8 formats currently handled by the Qwen3.5 MoE loader, such as block FP8 and ModelOpt-style FP8.

Current behavior

FreeToken already has support for the Qwen3.5 MoE model architecture, so no new model implementation should be required.

However, the current quantization detection and weight-loading paths do not appear to recognize compressed-tensors per-channel FP8 checkpoints.

As a result, this checkpoint cannot be loaded using its intended FP8 representation.

Requested support

It would be useful to add a dedicated loading path for compressed-tensors FP8 checkpoints with per-channel weight scales.

A possible initial scope could be:

1. Detect compressed-tensors FP8 checkpoints with float-quantized format.

2. Support FP8 tensors accompanied by per-channel weight_scale tensors.

3. Correctly handle scale broadcasting according to the stored tensor shape rather than assuming per-tensor or block scaling.

4. Support Qwen3.5 MoE expert weights, including stacked expert banks.

5. Respect the checkpoint's quantization ignore/exclusion rules.

6. Support weight and scale tensors located in different safetensors shards.

Possible incremental implementation

A first implementation does not necessarily need native per-channel FP8 MoE execution.

For correctness-first support, FreeToken could initially load:

FP8 weight + per-channel scale
              |
              v
        dequantize to BF16
              |
              v
     existing BF16 operators /
        MoE expert backend

This would allow the checkpoint to run without requiring a new FP8 MoE kernel immediately.

Native storage and execution could then be added separately:

FP8 expert weights
+ per-channel scales
+ dynamic activation scaling
        |
        v
native FP8 MoE expert backend

That would preserve the memory savings of the original checkpoint.

Likely affected areas

The main changes appear to be around the existing Qwen3.5 MoE quantization and weight-loading code, for example:

python/freetoken/models/qwen3_5_moe/config.py
python/freetoken/models/qwen3_5_moe/weight.py

The existing model architecture, attention implementation, and registry should not need major changes.

Suggested tests

Possible regression coverage:

• Detect compressed-tensors per-channel FP8 config correctly.
• Load a synthetic FP8 weight with a 1D per-output-channel scale.
• Verify dequantized weights against a reference implementation.
• Verify weight and scale lookup across different safetensors shards.
• Verify stacked MoE expert loading.
• Compare logits against a Hugging Face / reference implementation on a small deterministic input.

Motivation

FreeToken already supports the underlying Qwen3.5 MoE architecture, and supporting this additional FP8 checkpoint format would make models such as Ornith-1.5-35B-A3B-FP8 usable without requiring a separate architecture implementation.

It may also benefit other Qwen3.5 MoE checkpoints distributed using the same compressed-tensors per-channel FP8 format.