fix(agent): stop sending the reasoning prefill to native-tools chat providers - #287
Merged
Merged
Conversation
…roviders The trailing `<think>` prefill (and Gemma turn-framing tokens) are llama-server text-completion artifacts: the local template expects the open tag pre-typed at the generation point. On the native-tools chat transport the same prompt ships as a chat message to an OpenAI- compatible endpoint, where the literal tag is at best noise the model echoes back — and at worst corrupted server-side: Ollama Cloud mangles literal `<think>`/`</think>` strings in message content (ollama/ollama#17248), the trigger for #283. The injection only fired in hybrid configs (a local llama-server probing a think-tag model while completions route to a cloud provider), but there it also mis-parsed clean cloud replies: `normalizeContent` re-prepended the open tag and the stream parser started pre-opened, so a reply that never emitted `</think>` was swallowed whole as reasoning. - build-prompt: new `suppressReasoningPrefill` input drops the trailing reasoning prefill, the Gemma turn framing, and the reasoning system token; the step executor sets it for `toolTransport: "native_tools"`. - step-executor: parsing no longer assumes a prefill that was not sent (`promptCarriesReasoningPrefill` / `completionAssumesOpenReasoning`); grammar-parsed completions keep the legacy prepend — the GBNF prelude root structurally starts mid-think — including on cross-transport fallover. The one-shot repair prompt stops re-appending `<think>`. - profile-invariants: `checkProfilePromptAligned` learns the suppressed shape (the prompt must NOT end with a reasoning prelude). - provider-presets: document the upstream Ollama Cloud corruption next to the preset. The issue's proposed blanket rewrite of thinking-tag strings in all outgoing content for ollama.com endpoints is deliberately NOT implemented: silently mutating user text and tool results is the same silent-corruption class relocated client-side, and the server-side half is Ollama's bug to fix. Fixes #283 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-transport fallover Adversarial review of the prefill-suppression fix (#283) found a real streaming regression in the documented default hybrid chain — cloud (native_tools) primary with a grammar local last resort (`appendLocal`) on a think-tag profile: - consumeStream keyed `preOpenedThink` off the PRIMARY transport, but a grammar-served fallover stream starts mid-`<think>` (the GBNF prelude root emits `body "</think>"` with no open tag), so the parser silently swallowed the reasoning text — no live `reasoning_delta`s for every sticky-override turn of an outage, where main surfaced them. The `servedTransport` stamp on the stream's return value arrives only after the last delta, too late to reconfigure a parser. Fixes, in dependency order: - completion-types: `StreamChunk.servedTransport` — the fallback streamer seam now stamps the serving link's transport on EVERY chunk, not just the final result, so live consumers can adapt up front. - step-executor.consumeStream: the stream parser is created lazily off the first chunk's stamp (primary transport when unstamped, i.e. the direct non-fallback path), restoring live reasoning classification for grammar-served fallover streams. - llm-fallback-seam: per-link prompt substitution. The main prompt for a native-tools primary is prefill-suppressed, which handed the grammar fallover link a prompt/template mismatch (GBNF still forces mid-think output). `LlmStreamParams.grammarPrompt` carries a lazy, memoized prefill-carrying variant built only when a grammar link is actually chosen; the one-shot repair retry rebuilds both variants repair-shaped so a fallover retry never sees the stale base prompt. - completionAssumesOpenReasoning now keys purely off the served/parse transport: grammar-served output always continues an open think block; a chat completion never does. This also stops the (documented- unsupported) grammar-primary -> native-link ordering from swallowing a clean chat reply whole as reasoning; the literal prefill still shipping to the chat link in that reverse ordering remains, matching AGENTS.md's "order native-tools links at or above the first grammar-only link". - profile-invariants: the prefill-suppressed branch also flags leaked Gemma turn-framing tails, not just reasoning open tags. Coverage for the pinned invariant 8 gap that let this slip past CI: fallback-e2e now exercises think-tag profiles through the REAL seam factories, unary and streaming (live deltas + per-link prompt shape), llm-fallback-seam.test pins the per-chunk stamp and the grammarPrompt substitution, and step-executor.test pins both fallover directions. All 8 new tests fail without the src changes (verified by stashing). lint clean; agent+prompt+llm 877/877; runtime+tui/providers 336/336; full suite 6447/6448 — the one failure (sidecar/send-message-concurrency) fails identically on pristine origin/main in a clean worktree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stops the agent from injecting the local-model reasoning prefill — the trailing
<think>open tag (qwen-think) and the Gemma turn-framing tokens — into prompts served over the native-tools chat transport, and makes response parsing stop assuming a prefill that was never sent.Why
The prefill is a llama-server text-completion artifact: the local template expects the open tag pre-typed at the generation point. On
toolTransport: "native_tools"the same prompt ships as a chat message to an OpenAI-compatible endpoint, where the literal tag is at best noise the model echoes back — and at worst corrupted server-side: Ollama Cloud mangles literal<think>/</think>strings in message content (ollama/ollama#17248, ollama/ollama#17617), which is what #283 reports.Scope note: contrary to the issue's "every request" claim, a pure Ollama-Cloud setup was never affected — think profiles come only from the local llama-server
/propsprobe, so cloud-only runs useplain-instructand inject nothing. The injection fired in hybrid configs (managed local llama-server probing a think-tag model while completions route to a cloud provider). In those configs there was a second, latent bug this PR also fixes:normalizeContentre-prepended the open tag and the stream parser started pre-opened, so a clean cloud reply that never emitted</think>was silently swallowed whole as reasoning — affecting every cloud provider, not just Ollama.How
build-prompt: newsuppressReasoningPrefillinput drops the trailing reasoning prefill, the Gemma turn framing, and the reasoning system token; the step executor sets it whentoolTransport === "native_tools".step-executor:promptCarriesReasoningPrefill/completionAssumesOpenReasoninggate every consumer that assumed "the open tag was already sent" —normalizeContent,resolveReasoning,replyFallbackBatch, the stream parser'spreOpenedThink, and the one-shot repair prompt (which no longer re-appends<think>). Grammar-parsed completions keep the legacy prepend — the GBNF prelude root structurally starts mid-think — including on cross-transport fallover to a grammar local link.profile-invariants:checkProfilePromptAlignedlearns the suppressed shape (such a prompt must NOT end with a reasoning prelude).provider-presets: documents the upstream Ollama Cloud corruption next to theollama-cloudpreset.What this deliberately does NOT do
The issue's proposed blanket string-replacement of
<think>/</think>/【思考】in all outgoing message content forollama.comendpoints is not implemented. Silently and lossily rewriting user text and tool results is the same silent-corruption class the issue complains about, relocated into the client — it would break any task where the agent must reproduce file content verbatim. The server-side half is an upstream Ollama serving bug for Ollama to fix; if maintainers still want a client-side mitigation it should be an opt-in provider flag with placeholder round-tripping, i.e. a separate design discussion.Test evidence
build-prompt.test.ts; native-tools prompt hygiene + "reply not mis-parsed as reasoning", stream deltas not reclassified as pre-opened reasoning, and repair-prompt hygiene instep-executor.test.ts. Plus a grammar-transport regression case pinning legacy behavior (prefill still sent, reasoning still extracted).npm run lint(tsc --noEmit) clean.src/sidecar/send-message-concurrency.test.ts) fails identically on pristineorigin/main(verified in a separate worktree) and is unrelated.Fixes #283