Tags: SuperLLMModel/forge
Tags
v0.7.0: lineup refresh + tool-error channel + docs sweep + MODEL_REGI… …STRY (antoinezambelli#77) * Add Qwen3.6 and Qwen3-Coder GGUF-stem sampling alias entries The v0.6.0 sampling defaults map had ollama-tag entries for these two models but no GGUF-stem aliases. Llamafile-backed eval consumers (forge-code) use the GGUF stem as canonical identity and need recommended_sampling=True to resolve, which raised UnsupportedModelError without the aliases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * AMD sysfs probe in detect_hardware() to fix silent 4K Ollama fallback (antoinezambelli#61) Adds a probe ladder (nvidia-smi -> /sys/class/drm AMD sysfs) and a one-shot WARN when both fail, so AMD unified-memory rigs no longer silently land on the 4096-token Ollama tier. ROCm tooling stays out of scope per the Vulkan/RADV decision. HardwareProfile gains gpu_vendor and memory_kind (discrete | unified) so downstream callers and logs can distinguish a 64GB unified pool from 64GB of discrete VRAM. Defaults preserve existing call sites; llama- server / llamafile path is unchanged (it reads /props from the running server and doesn't go through detect_hardware()). Verified on rig-03 (Corsair AI 300, Strix Halo): detect_hardware() returns memory_kind='unified', vram_total_gb=64.0, which lands cleanly in the >=48GB Ollama tier. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chat_template_kwargs plumbing + Qwen3-Next/Mistral-Small-4 sampling defaults llama-server's OpenAI-compat /v1/chat/completions accepts a top-level chat_template_kwargs dict that gets unpacked into Jinja chat-template variables (reasoning_effort, enable_thinking, etc.). Previously not exposed via forge — both LlamafileClient._apply_sampling and the proxy's _extract_sampling allowlists dropped the field. Both allowlists extended and a chat_template_kwargs constructor param threaded through LlamafileClient. Whole-value replacement at the field level (no nested merge) — the simple shape covers our current use case; revisit if a caller ever needs to override one nested key without losing the rest. New sampling-registry entries for the big-models lineup (rig-03): - Qwen3-Next-80B-A3B-Instruct: T=0.7, top_p=0.8, top_k=20, min_p=0 (HF card thinking-mode profile). - Qwen3-Coder-Next: T=1.0, top_p=0.95, top_k=40 (notably different from Qwen3-Coder-30B; card omits min_p/repeat_penalty). - Mistral-Small-4-119B-2603: T=0.7 + chat_template_kwargs={"reasoning_effort": "high"} per HF card guidance for high-effort reasoning runs. Validated end-to-end via rig-03 smoke — Mistral emitted 776-char reasoning trace on the first probe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Mistral [TOOL_CALLS] bracket-tag tool-call rescue parser Devstral-Small-2 and Mistral-Small-3.x emit tool calls in their training- data native format when prompted via prompt-mode: [TOOL_CALLS]<name>{args}, with optional whitespace or newline between the name and the opening brace. forge's existing rescue parsers (standard JSON, rehearsal syntax, Qwen XML) all miss this shape, so calls were dropping into ToolCallError after 5 nudge retries. Empirically across rig-03 v3-big runs, ~19/26 of Devstral's retries- exhausted failures were [TOOL_CALLS]-format emissions (12 view, 3 bash, 2 glob, 1 view bare, 1 grep) — same pattern shows up in Mistral-Small-3.2 in mid-conversation apology-then-bracket exchanges. New _parse_mistral_bracket_tool_calls hooked as Strategy 4 in rescue_tool_call (after JSON, rehearsal, Qwen XML). Brace-balance scan with string-literal awareness handles nested objects and escaped quotes inside arg values. Eleven new tests in TestMistralBracketRescue cover the no-separator / whitespace / newline variants, nested braces, escaped quotes, multi-call lines, unknown-tool rejection, unclosed-brace fall-through, JSON-precedence, and bracket-after-thinking. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Multi-shard GGUF model-key fix + gpt-oss-120b/Nemotron-3 sampling defaults LlamafileClient: strip "-NNNNN-of-NNNNN" shard suffix from gguf_path.stem so multi-shard GGUFs (e.g., Mistral-Small-4-119B-2603-UD-Q4_K_M-00001-of- 00003.gguf) resolve to the canonical model name in MODEL_SAMPLING_DEFAULTS. Single-shard stems are unaffected. sampling_defaults: add registry entries for gpt-oss-120b and NVIDIA- Nemotron-3-Super-120B-A12B (both ollama-tag + GGUF-stem variants): - gpt-oss-120b: T=1.0, top_p=1.0, top_k=0, min_p=0.0, chat_template_kwargs={"reasoning_effort": "high"}. Three discrete reasoning levels (low/medium/high). Per llama.cpp guide, do NOT set repetition penalties (registry omission == None == field dropped). - Nemotron-3-Super-120B-A12B: T=1.0, top_p=0.95, chat_template_kwargs={"enable_thinking": True, "force_nonempty_content": True}. Hybrid Mamba-2 + Transformer + MoE. Three reasoning states (off, on+low_effort, on). Defaulting to full effort + force_nonempty_content per coding-agent serving guidance. Native tool format is Qwen3-coder- style; existing forge prompt-mode rescue parsers should suffice. Both default to high/full-effort reasoning to mirror Mistral-Small-4 — bring down once eval signal is in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Surface step-enforcement and prereq violations as tool errors, not user nudges The two enforcement paths in WorkflowRunner — premature terminal (3b) and prerequisite violation (3b.2) — previously emitted a USER message after the assistant tool_call placeholder to nudge the model toward correct behavior. This shape interacts badly with two other forge mechanisms: 1. _merge_consecutive folds consecutive same-role plain user/assistant messages to satisfy Mistral's parity checker. The nudge user message gets folded into the original user input, hiding the correction signal inside a longer message body. 2. On thinking-mode models served via llama-server with --jinja (gpt-oss harmony, Qwen3.5 with reasoning, etc.), trailing-assistant requests are rejected as "Assistant response prefill is incompatible with enable_ thinking" — but the broader observation here is that trailing-assistant tool_call requests during enforcement violations were also producing ambiguous wire shapes the model didn't pretrain on. Net effect with v4 forge-code (apply_pytest_first / apply_ruff_first mechanism): gpt-oss-120b reliably violated prerequisites three turns in a row before exhausting and raising PrerequisiteError. Wire trace showed the model receiving [system, user(merged with nudges), assistant(tc), assistant (tc)] — no signal for "your prior call was rejected." This commit replaces the trailing user nudge with one tool-error message per tool_call in the batch. The wire shape becomes [..., assistant(tc), tool([PrereqError] You must call X first)] — the canonical "tool failed, try again" pattern that all OpenAI-tool-trained models handle natively. Validated on rig-03 with gpt-oss-120b on v4: 18-row B1+F1+F3 slice, 887 wire requests, zero 400s, zero PrereqError exhaustions, 17/18 rows complete through the full specialist chain (1 ERROR was a MaxIterationsError on F3, unrelated to this patch). Should be safe across the model fleet — this just changes which message role the same nudge content arrives on. Quick regression on rig-02 recommended before merging widely. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Surface unknown-tool retry as tool error, not user nudge Mirrors the dogfood patch (04852a4) for the third remaining nudge path: when ResponseValidator detects a tool_call with an unknown tool name, the runner used to emit assistant(tool_call) followed by a trailing user-role nudge. Now it emits one TOOL message per tool_call, prefixed with [UnknownTool], tied to the original tool_call_id. Same rationale as step-enforcement and prerequisite violations: corrective signal rides the canonical channel the model was pretrained on (tool_call → tool_result), which survives heavy-context attention drop-off and Mistral _merge_consecutive folding. The bare-text retry path (model emitted no tool_call at all) intentionally stays as user-role nudge — no tool_call_id exists to anchor a tool error to. Validated against forge-code v4 eval (Qwen3.6 / Coder / Devstral, N=5 full suite, 405 rows): ToolCallError exhaustions dropped from 24 to 11, confirming models now recover from unknown-tool emissions instead of dying at the 5-strike validator counter. Net error count unchanged (108 → 111) but failure-mode locality is correct: PrereqExhausted on Coder rose by 10, exposing that Coder's deep issue is tool-ordering, not validator exhaustion (the validator was masking it). Diagnostically better. All three enforcement paths in the runner/inference layer now use the tool-error channel; only bare-text retry remains on user-role, by necessity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Sampling defaults: Nemotron-3-Nano-30B-A3B-Q4_K_M NVIDIA Nemotron-3-Nano-30B-A3B (Mamba-2 + Transformer hybrid + MoE, 30B total / ~3.5B active). Card splits sampling into two presets: - Reasoning: T=1.0, top_p=1.0 - Tool-calling: T=0.6, top_p=0.95 (selected here) Empirical context (forge-code v4 eval, N=135 prompt mode rig-02): the Reasoning preset's wider nucleus drove 16% deception on F3-class scenarios in an earlier N=5 probe; the deterministic preset cleared that to 0% deception across the full sweep, at the cost of higher hard-fail rate (48%) — the model becomes never-silently-wrong but more fragile under v4's specialist-pipeline gates. Native tool format is qwen3_coder XML — irrelevant for forge prompt-mode runs. * Sampling defaults: Qwen3.5-122B + gpt-oss/Nemotron-Super tweaks + server timeout bump - Add Qwen3.5-122B-A10B-Q4_K_M entry (T=0.7, top_p=0.8, top_k=20 — card's balanced preset, starts in instruct mode via --reasoning-budget 0) - gpt-oss-120b: reasoning_effort high → medium - Nemotron-3-Super-120B: add low_effort=True for lighter thinking - server.py: _wait_healthy timeout 120s → 180s for big-MoE cold start * v0.7.0: small-model lineup refresh + eval dataset Lineup (rig-01 small-model tier, ≤14B): - Cut: llama3.1 (Q4/Q8), mistral-7b-v0.3 (Q4/Q8), mistral-nemo (Q4), granite-4.0 (h-micro/h-tiny Q4/Q8). All bare <30% — too weak to be informative; superseded by 4.1 / E4B / phi-4. - Add: granite-4.1-8b (Q4/Q8), gemma-4-E4B-it (Q4/Q8), phi-4 (Q4). batch_eval: - _PROMPT_ONLY_MODELS: skip native FC for models lacking training for OpenAI tool_calls schema. phi-4 verified via curl 2026-05-14. - _NO_RECOMMENDED_SAMPLING_MODELS: recommended_sampling=False for models with no formal sampling guidance from any official source. - recommended_sampling threaded per-config through _build_client for ollama / llamaserver / llamafile backends. sampling_defaults: - Granite 4.1 (8b Q4/Q8) entries — mirror granite-4.0 IBM convention; marked unconfirmed pending IBM publication. - Phi-4 base documented in 'Intentionally absent' block. Dataset: eval_results_rig-01_v0.7.0.jsonl (74 cells × 1300 = 96,200 rows). Patches evaluated: 04852a4 (prereq/step → tool-error channel), 0ca3eb0 (unknown-tool retry → tool-error channel). Apples-to-apples delta on 21 common configs vs v0.6.0: +0.7pt overall, -1.2pt advanced_reasoning — both within CI. Published-leaderboard floor lifts +16.9pt via composition (weak-model cuts). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Docs sweep: README/USER_GUIDE/BACKEND_SETUP/WORKFLOW + surgical ARCHITECTURE patches README: - Open with the contract (any tools, any order; structure is opt-in via required_steps/prerequisites/terminal_tool) before the eval pitch. - New "What forge isn't" section: not an agent orchestrator, not a coding harness (proxy-mode pointer for harness users). - Three-ways list reordered with proxy first (most popular entry point). - Quick Start swapped Ollama -> llama-server; split into "start the server however you do" + "the Python you'll run" blocks so neither half makes environment assumptions. - Proxy section expanded with explicit fortifies / does-not list, backend compatibility notes (managed=llamaserver|llamafile|ollama; external=any OpenAI-schema endpoint), and a useful-flags table. - Project Structure tree adds guardrails/guardrails.py and proxy/__main__.py. - Eval Harness section surfaces report.py --html and --markdown flags. BACKEND_SETUP: full rewrite. ~360 -> ~135 lines. Cut model-pick prose, Reasoning-vs-Instruct comparison, Windows-only install instructions, Ollama Modelfile tutorial, llamafile distribution-format explainer, "Run the eval" subsections, VRAM tables, and standalone smoke-test section. Kept boot command + flag table + curl smoke-test + forge client snippet per backend, plus the reasoning-budget gotcha. Added Anthropic section using the [anthropic] extra install path. USER_GUIDE: - Single-Turn "What happens under the hood" notes that step/prereq violations surface on the tool channel post-v0.7.0; bare-text retry nudges still user-role. - Tool Prerequisites section rewritten to reflect the tool-error wire shape ([PrereqError] prefix) and how that's friendlier to OpenAI-tool-trained models. - Choosing a Backend table cut (duplicate of BACKEND_SETUP); replaced with pointer. WORKFLOW: agentic-loop flowchart renames NUDGE/UNKNOWN_NUDGE/STEP_NUDGE/ PREREQ_NUDGE to *_TOOL_ERROR to match v0.7.0 tool-error channel emission; compaction table updates step_nudge and prerequisite_nudge role to tool, retry_nudge stays user; bfcl/ removed from module diagram (directory was removed; ADR-009 retained as historical artifact). ARCHITECTURE: surgical signature patches only. - MessageType enum: add PREREQUISITE_NUDGE, CONTEXT_WARNING; tool-error channel noted on STEP_NUDGE / PREREQUISITE_NUDGE comments. - WorkflowRunner.__init__: add retry_nudge kwarg. - WorkflowRunner.run(): add cancel_event kwarg; Raises includes WorkflowCancelledError, PrerequisiteError, StepEnforcementError. - CompactStrategy.compact(): trigger_tokens -> budget_tokens; docstring reflects "strategy owns its own threshold logic". - ContextManager.__init__: drop compact_threshold; add context_thresholds + on_context_threshold with docstrings. - LlamafileClient.__init__: add top_p, top_k, min_p, repeat_penalty, presence_penalty, chat_template_kwargs, slot_id. Numbers in MODEL_GUIDE/EVAL_GUIDE deliberately untouched -- pending v0.7.0 report.py regeneration. ARCHITECTURE.md structural rebuild deferred to a separate commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Rebuild ARCHITECTURE.md as a Design Decisions doc Cut everything that just restated source: class signatures, dataclass field lists, exception hierarchies, client-adapter implementation walks, data-flow prose, file-structure tree, test-strategy section, server- management deep dive, "consumer validation" case studies (those were NORA/personal-project framings, not forge architecture). What earns its keep in the rebuild: - Design Principles (5) with concrete consequences linked to ADRs - Surface Modes (WorkflowRunner / proxy / middleware) — the three-way shape with the shared guardrails-as-foundation diagram - Guardrails table: what each one catches and why it exists — including the v0.7.0 tool-error-channel rationale (rationale layer was missing this; USER_GUIDE explains operationally, WORKFLOW shows in the diagram, ARCHITECTURE now explains why it's shaped this way) - Compaction priority table + the "reasoning survives Phase 2" rationale - Synthetic respond tool — the bare-text-eval rationale, ADR-013 pointer - Sampling defaults opt-in semantics — ADR-014 pointer - Where-to-find-things navigation footer (source is authoritative for signatures; WORKFLOW.md for diagrams; decisions/ for ADRs; this doc for the why) 1701 -> 165 lines. The reader pact is now: "WORKFLOW.md if you want the shape; ADRs if you want past decisions in detail; source if you want exact APIs; this doc if you want the why behind forge's shape." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.7.0: docs sweep, eval dataset rename, numbers regen, MODEL_REGISTRY Version bump 0.6.0 -> 0.7.0 (pyproject.toml; __init__ uses importlib.metadata). Full CHANGELOG entry added. Eval datasets renamed to versioned filenames (LFS-tracked): eval_results.jsonl -> eval_results_v0.6.0.jsonl eval_results_rig-01_v0.7.0.jsonl -> eval_results_v0.7.0.jsonl Both files remain in the repo for cross-version comparison. Numbers regen: report.py against v0.7.0 dataset produced fresh: - docs/results/dashboard.html (React build, self-contained) - docs/results/raw/reforged/{all,by-family,by-backend}.md - docs/results/raw/reforged-vs-bare.md - docs/results/raw/native-vs-prompt.md MODEL_FAMILIES (report.py): added granite-4.1-8b (Q4/Q8) and phi-4-Q4_K_M entries so cross-backend rollups in by-backend.md group correctly. New: docs/MODEL_REGISTRY.md Three tiers: Current (in v0.7.0 eval, 9 model families + Claude trio carrying v0.6.0 numbers), Retired (5 models cut from current eval -- bare <30%), Unpublished (17 entries with sampling params staged but no published eval). All values pulled from sampling_defaults.py with source URLs + chat_template_kwargs annotations where relevant. Rewrites: - MODEL_GUIDE.md (333 -> 145 lines): trimmed to opinions + rationale. Leaderboard reproduction moved to dashboard / markdown views. Native- vs-prompt heuristic corrected based on v0.7.0 data (not workload- driven; sensitivity is per-family). EVAL_GUIDE.md: new "Committed datasets" subsection documenting the versioning convention. README.md: eval-pitch numbers updated to v0.7.0 (84% top local + Sonnet 85%->98% qualified as v0.6.0 measurement). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>