Support muse glimmer - #4
Conversation
… NVFP4) Dense 30B text tower of the multimodal wrapper, served text-only like the other VL checkpoints. Architecture-wise it is a mix of pieces we already run -- gemma4's hybrid SWA/full layer split and qwen3.5's gated attention + CT-NVFP4 dense path -- plus a few new wrinkles: - [SWA x3, full] pattern where the full layers are NoPE (layer_rope_theta 0): rope is skipped entirely on those layers, guarded so base=0 never reaches RotaryEmbedding - weightless per-head qk RMSNorm with an extra query scale (3.87); the scale commutes with rope so it folds into AttentionSpec.sm_scale - separate self_attn.gate_proj (sigmoid output gate), fused into the q/k/v projection since it reads the same normed layer input - centered (1+w) sandwich norms with two eps (1e-5 pre / 1e-8 post), applied at runtime like M3's gemma norms, never baked into bf16 - NormedEmbedding (weightless RMSNorm after embed) and gemma-style logit softcap with a pre-scale (output_multiplier) The RedHatAI NVFP4 release is llm-compressor W4A16 on every text Linear (attn gate included), same layout as Qwen3.6-27B-NVFP4 -- the CT helpers moved from qwen3_5_moe/weight.py into models/loader.py so both models share them, and the W4A16 kernels are reused unchanged. Server side the model speaks the ATEM channel protocol: to=self segments are reasoning, to=user is content, tool channels carry <atem:function_calls> invoke/parameter XML (string values not trimmed, per the template's own contract). New reasoning parser + detector handle the bare first segment (generation resumes after <|start|>assistant), and reasoning_strength low/medium/high/xhigh rides the thinking gears, with OpenAI reasoning_effort mapped onto it.
AutoConfig refuses to load a config.json whose model_type it does not know when the checkpoint ships no auto_map (Muse-Glimmer needs transformers >= 5.15; MiniMax-M3 only loaded because nvidia bundled a remote config class). FreeToken never instantiates modeling code -- parse_config just reads fields -- so fall back to an attribute view over the raw JSON, wrapping *_config sub-dicts the way PretrainedConfig nests them and leaving everything else (rope_parameters etc.) as plain dicts.
The real RedHatAI checkpoint splits layer 49's down_proj across the shard boundary: weight_packed sits in shard 1, its weight_scale/global in shard 2, so reading siblings from the same file handle blew up mid-load. Route all sibling lookups through a lazy shard-map reader keyed off the safetensors index. Also pins the loaders with synthetic-checkpoint roundtrips: bf16 keys/fusion order against the model's state_dict, and the native-FP4 path with cross-shard scales, reciprocal globals and dropped activation scales.
…mplate The jinja chat template owns every special token (HF's apply_chat_template tokenizes with add_special_tokens=False for the same reason), but the templated path re-encoded with the defaults, so tokenizers that auto-add bos (muse-glimmer, llama lineage) got a doubled bos at position 0. Found by greedy-parity against a pure-torch Muse reference: 70 prompt tokens vs the reference's 69, and a diverging generation. With the fix the served greedy decode matches the reference token for token through the full turn. The dsv4 encoder path keeps its existing behavior.
|
Review + e2e done: BF16 greedy matches transformers 5.15 char-for-char on H100; NVFP4, streaming, and all test suites pass; model math and weight loading are clean. Parsers were cross-checked against vLLM's HIGH1. Channel close mid-invoke corrupts the next call ( 2. Headerless channel switch drops the call. The model sometimes leaves MED
LOW
Echo-becomes-execution riskFT executes |
…ncation, execution scoping Review findings, cross-checked against vLLM's muse_glimmer parsers: - A channel closer arriving mid-invoke now finalizes the open call: the streamed argument JSON is closed (never emit broken arguments), completed parameters land in the ledger, and the ordinal advances so the next channel's call can't merge into it. Truncated channels log a warning instead of vanishing silently. - A complete headerless "to=<name><|message|>" is a segment boundary in both parsers (vLLM's rule) -- the model leaves to=self without <|eom|> for empty-argument calls, which used to absorb the whole tool channel into reasoning. Committing to the bare stream-start header now also requires a FULL match (with a length bound), so content that merely starts with "assistant"/"to=" streams as text instead of being eaten; the verbatim <|start|>assistant normalization became unnecessary and is gone. - ATEM markup is executed only inside a tool-recipient channel: a block quoted in a to=user body -- or the system prompt's own ATEM example echoed back -- renders as text instead of becoming a real call (vLLM's scoping). - Template-doubled recipients (a bare-name tool renders as name.* and the model emits get_weather.get_weather) collapse to the registered head, via a new name-normalization hook on InvokeParamStreamMixin. - detect_and_parse is now a replay of the streaming machinery, so both paths share one definition of channel classification (a raw to=self body never leaks into content with --reasoning-parser off), execution scoping, and value semantics (the one-shot lookahead regexes that truncated values containing literal </atem: tags are gone with them). - The preserved tool-channel slice holds back partial markers while streaming, so emitted content can no longer grow a "<|start|" that the next chunk reveals to be the next segment's opener. - RawConfigShim serves _name_or_path through the underscore guard (DSV4's parse_config reads it to find inference/config.json). Re-validated on the real NVFP4 checkpoint: greedy parity prompt unchanged, bare-name and namespaced tool calls, streaming argument JSON valid.
|
Round 2, on HIGH1. Exiting a tool channel without a terminator drops the rest of the turn (streaming) ( 2. max_tokens mid-invoke still ships invalid JSON arguments ( 3. Non-streaming drops the second invoke block in one channel ( 4. MED5. 6. End-of-stream flush drops held text instead of delivering it. Two confirmed shapes, one fix: (a) a complete response that merely looks like a bare header prefix ( Small
|
… truncation, O(n) streaming
Second review pass on the ATEM parsers, plus the shared-infra cleanups it
flagged:
- A tool channel that ends without its own terminator (abutting <|start|>
header or headerless to=user switch) now reaches the detector as a
delimited block: the reasoning parser appends a synthetic <|eom|> to the
preserved slice, so the rest of the turn streams instead of being
discarded in tool mode.
- max_tokens mid-invoke: a finalize_stream hook runs at end-of-stream (the
same closing-fragment logic the channel boundary uses), so the client's
concatenated argument fragments end as valid JSON -- verified on the wire
at several truncation points, including mid-string. unstreamed_arguments
also stops treating the {} ledger entry as absent.
- The tool-mode boundary transition loops the mixin until it is inert
before consuming the closer: a second invoke block in the same undrained
buffer (always the case for the one-shot replay) parses instead of
vanishing as markup debris.
- reasoning_strength counts as a thinking kwarg only for muse_glimmer;
globally it swallowed every other family's toggle mapping when a gateway
forwarded muse-style kwargs.
- The reasoning parser streams by consuming its buffer (the full re-scan
per chunk was quadratic and stalled the serving loop on this family's
long default CoT), and end-of-stream flush now delivers held text --
a reply that merely looks like a bare-header prefix, or prose stranded
after a literal closer -- instead of dropping it. Discarded tool-channel
prose leaves a warning.
- detect_compressed_tensors_nvfp4 gates on group_size 16 + tensor_group
(vLLM's rule); an MXFP4 checkpoint fails with a clear unsupported-scheme
error instead of a shape assert inside the NVFP4 loader.
- Dedup: ShardReader and the CT scale suffixes moved to models/loader.py
and qwen3_5_moe's CT path now resolves scales through the shard index
too (same layer-49 splitting hazard); the quant-linear factories moved
to models/quant_linear.py, replacing muse's copy.
Main's #5 replaced the per-family thinking-gear registry with checkpoint probing (tokenizer/effort.py), which supersedes the muse entries this branch had added to model_meta -- resolved by taking main's side and mapping muse onto the new machinery where it belongs: the render layer now broadcasts reasoning_effort as reasoning_strength too (the same every-spelling rule the thinking toggles use; Jinja ignores undeclared variables and an explicit caller spelling wins). Through the broadcast the probe sees muse's template grade effort (validating nothing, defaulting high), so /v1/cache/status derives the OpenAI gear triple with default high, and muse's native xhigh still passes quantization untouched. The double-bos fix is re-applied to the reworked tokenize path: template-rendered prompts encode with add_special_tokens=False, raw strings and the dsv4 encoder keep the default.
|
Round 3, on MED1. Item-5 residual: "seek" mode is still O(n²), holds an unbounded buffer, and stops streaming entirely ( 2. Seek-mode EOS still drops text around a complete 3. The merge dropped the branch's effort folding: out-of-vocabulary efforts reach the model verbatim ( 4. A literal channel-marker inside a parameter value splits the call — and the remainder now surfaces as raw markup in user content ( Small
|
…t vocabulary, channel residue
Third review pass, all on the ATEM parser edges and the effort merge residue:
- The reasoning parser's seek mode (between segments) now streams eagerly as
content with the standard hold-back and consumes its buffer -- the
hold-until-flush behavior was still quadratic and sat on a headerless
continuation (a post-<|eom|> repetition loop) for the whole stream. Seek
deliveries strip redundant closer tokens (the detector's rule), the
one-shot passthrough shortcut routes closer-containing text through the
replay so both paths agree, and a complete <|start|> whose <|message|>
never arrives is bounded: text before it is content immediately, the
candidate is held only to a plausible header span (released as literal
text past it, capped debris at EOS) instead of eating the rest of the turn.
- Out-of-vocabulary efforts no longer reach the model verbatim: a
non-validating grader is served its dialect's ladder instead of the whole
scale -- the probe now also learns WHICH spelling the template reads, so
the reasoning_strength dialect (muse; its card documents xhigh) gets
low/medium/high/xhigh while plain effort graders keep the OpenAI triple
(the gpt-oss-shape pin stays intact). minimal quantizes to low, max to
xhigh, none falls to the template default; /v1/models and the
/v1/cache/status gears both advertise the served vocabulary, xhigh
restored to the gear list.
- A channel boundary that fires mid-invoke marks the channel truncated:
until the next boundary the detector routes incoming text (the broken
channel's raw ATEM residue) through the dropped-prose path instead of user
content -- a literal to=user<|message|> inside a parameter value no longer
leaks "</atem:parameter>..." into the visible reply. Dropped channel prose
is accumulated and logged once per channel rather than per fragment, and
the detector mirrors the header-span bound in text mode.
- detect_compressed_tensors_nvfp4 collects verdicts across ALL config groups
before returning: a mixed {nvfp4, mxfp4} checkpoint now raises the clean
unsupported-scheme error in either key order.
|
Round 4, on HIGH1.
The mark is also set when the boundary that fired was itself a closer (the channel ended; there is no residue to guard against). Fix: don't gate on "next boundary" — clear the mark immediately when the firing boundary was a closer, and for the abutting-start/inline-switch case drop by shape: consume only runs of ATEM closing markup ( MED2. The header-span cap is bypassed whenever a 3. Span release breaks protocol-legal long headers in streaming — trailing reply lost from span ~119, the tool call itself lost from span ~129 (
The release fires while 4. Deliver-then-swallow across layers at EOS ( |
…shape, span bounds unified The round-3 residue guard regressed on the production pipeline: the mark gating text into the dropped-prose path waited for a boundary that the pipeline never delivers (the reasoning parser terminates every preserved slice and unwraps everything else), so a single truncated invoke silently swallowed the rest of the turn's reply. The mark now clears by SHAPE: text mode consumes only runs of ATEM closing markup (closing tags + whitespace) and releases at the first non-markup character -- trailing tag debris still drops, a real reply flows immediately, and the mark deliberately survives the synthetic terminator because the pipeline delivers the residue after it. Production-order (_pipe) tests now cover both reviewer wires, plus a wire corpus (historical degenerate shapes x chunkings) asserting streaming/one-shot/pipeline agreement -- standalone-detector tests being green over a pipeline regression is the failure mode this file has now hit twice. The header-span bounds are unified across the remaining shapes: the bound applies whether or not a <|message|> is already in the buffer (a stray literal <|start|> + junk + the next segment's real header no longer parses as one giant header, which also let a to= inside the junk hijack the recipient and diverge one-shot from streaming); recipient names cap at 64 everywhere (matching the inline/bare-start rules, so both modes agree and longer names degrade identically); the release condition gains len(<|message|>) slack so a protocol-legal long-name header whose marker is mid-arrival isn't cut; and the detector's finish_streaming mirrors the capped-debris rule instead of blanket-dropping any residual containing <|start|> -- text the layer above deliberately delivered stays delivered.
|
Round 5, on
Item 4 is not fixed, and the EOS-discard family it belongs to got worse. The three items below share one root cause and one fix. HIGH1. A real MED2. Round-4 item 4 persists: the two layers still measure the header span on different byte streams, and the new LOW3. The One fix closes 1–3: at end of stream, a 4. A reply that literally begins with a complete ATEM closing tag is eaten after a truncated invoke ( Test integrityThe batch added production-order tests (good — the corpus does catch round-4's HIGH on
|
Round 5 landed on the root cause of the whole EOS-discard family: the two layers measure the header span on different byte streams (raw vs closer-stripped), so any capped-discard window at end of stream has edges that can never agree -- fixing one wire just moved the edge onto another. Remove the window instead of moving it: at end of stream a <|start|> candidate that never received its <|message|> is not a header. Both layers now deliver the tail and drop only the marker. This closes the R5 HIGH (reply after a stray start+junk segment dropped), makes the parser/pipeline span agreement trivial (R4 item 4), and removes the symmetric 129-139 drop window, all with one rule. Also per review: document the known cost of shape-based residue dropping (a reply that literally begins with a complete closing tag loses that tag), and repair the test pins that weren't holding weight -- the R3 residue trade is pinned on the original wire with exact expected output, the EOS agreement test now fails against the old blanket drop, the detector's giant-header bound gets its own pin, and the corpus checks closing-tag leaks and argument payloads, not just call names.
|
Round 6, on 1. (MED) A bare-header lookalike ending in Fix it by deleting the guessing layer instead of patching it: initialize both parsers from the prompt they continue. The chat generation prompt always ends with 2. Corpus test: add the shapes it exists to police. It passes 21/21 against 3. PIN3 docstring ( Mergeable from my side once item 1 lands. |
… the guessing layer The last member of the empty-turn family (round 6): a bare-header lookalike ending in <|message|> -- a recipient past the 64-char cap, a glued assistantto=x, or >8 leading newlines -- was delivered by the reasoning parser but blanked by the detector's end-of-stream debris rule. Every such bug came from the same place: both layers GUESSED whether the turn's first bytes were a bare header, each with its own regex and its own hold logic. Stop guessing. Every request that reaches these parsers is a templated chat generation whose prompt ends with <|start|>assistant, so the reasoning parser now starts header-open: it seeds a synthetic <|start|> and the turn's first bytes go through the same full-header machinery as every later segment (the same mechanism force_reasoning is for the <think> families, read from the prompt instead of re-derived). The detector takes the state as a constructor flag, on when it receives raw turn bytes without the reasoning parser stacked above. A junk recipient now yields the empty channel the model asked for; a header that never gets its <|message|> falls to the unfinished-header-at-EOS-is-text rule; the synthetic marker itself is never delivered. The bare-start regex, the undecided-hold classifier, the parser's "start" mode, the detector's stream-start commit and the <|message|> debris blanket all go away. One new rule the seed exposed: a channel header can never CONTAIN an ATEM control token, so a header candidate with a marker before its <|message|> is voided immediately in both layers -- turn-start prose ahead of the first real header flows as content instead of vanishing into a giant header, and the stray-start+junk shapes now resolve mid-stream instead of waiting for the EOS rule. Tests per review: the corpus gains the round-5 shapes it existed to police (stray start+junk, death inside an unfinished header, closing-tag residue) and its one-shot content check no longer hides behind an always-true disjunct; the giant-header pin's docstring now names the load-bearing assertion; new pins cover the three lookalike shapes, seed no-leak, and prose-before-first-header (each verified to fail under mutation). Re-ran the NVFP4 checkpoint end-to-end: chat, reasoning split, streaming and non-streaming tool calls, tool round-trip.
2 participants
No description provided.