Skip to content

fix(agent): stop sending the reasoning prefill to native-tools chat providers - #287

Merged
plombeer31 merged 2 commits into
mainfrom
fix/issue-283-ollama-cloud-thinking-tag
Aug 31, 2026
Merged

fix(agent): stop sending the reasoning prefill to native-tools chat providers#287
plombeer31 merged 2 commits into
mainfrom
fix/issue-283-ollama-cloud-thinking-tag

Conversation

@plombeer31

@plombeer31 plombeer31 commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

What

Stops the agent from injecting the local-model reasoning prefill — the trailing <think> open tag (qwen-think) and the Gemma turn-framing tokens — into prompts served over the native-tools chat transport, and makes response parsing stop assuming a prefill that was never sent.

Why

The prefill is a llama-server text-completion artifact: the local template expects the open tag pre-typed at the generation point. On toolTransport: "native_tools" the same prompt ships as a chat message to an OpenAI-compatible endpoint, where the literal tag is at best noise the model echoes back — and at worst corrupted server-side: Ollama Cloud mangles literal <think>/</think> strings in message content (ollama/ollama#17248, ollama/ollama#17617), which is what #283 reports.

Scope note: contrary to the issue's "every request" claim, a pure Ollama-Cloud setup was never affected — think profiles come only from the local llama-server /props probe, so cloud-only runs use plain-instruct and inject nothing. The injection fired in hybrid configs (managed local llama-server probing a think-tag model while completions route to a cloud provider). In those configs there was a second, latent bug this PR also fixes: normalizeContent re-prepended the open tag and the stream parser started pre-opened, so a clean cloud reply that never emitted </think> was silently swallowed whole as reasoning — affecting every cloud provider, not just Ollama.

How

  • build-prompt: new suppressReasoningPrefill input drops the trailing reasoning prefill, the Gemma turn framing, and the reasoning system token; the step executor sets it when toolTransport === "native_tools".
  • step-executor: promptCarriesReasoningPrefill / completionAssumesOpenReasoning gate every consumer that assumed "the open tag was already sent" — normalizeContent, resolveReasoning, replyFallbackBatch, the stream parser's preOpenedThink, and the one-shot repair prompt (which no longer re-appends <think>). Grammar-parsed completions keep the legacy prepend — the GBNF prelude root structurally starts mid-think — including on cross-transport fallover to a grammar local link.
  • profile-invariants: checkProfilePromptAligned learns the suppressed shape (such a prompt must NOT end with a reasoning prelude).
  • provider-presets: documents the upstream Ollama Cloud corruption next to the ollama-cloud preset.

What this deliberately does NOT do

The issue's proposed blanket string-replacement of <think>/</think>/【思考】 in all outgoing message content for ollama.com endpoints is not implemented. Silently and lossily rewriting user text and tool results is the same silent-corruption class the issue complains about, relocated into the client — it would break any task where the agent must reproduce file content verbatim. The server-side half is an upstream Ollama serving bug for Ollama to fix; if maintainers still want a client-side mitigation it should be an opt-in provider flag with placeholder round-tripping, i.e. a separate design discussion.

Test evidence

  • 5 new vitest cases, all of which fail without the fix (verified by stashing the src changes): qwen/gemma prompt-tail suppression in build-prompt.test.ts; native-tools prompt hygiene + "reply not mis-parsed as reasoning", stream deltas not reclassified as pre-opened reasoning, and repair-prompt hygiene in step-executor.test.ts. Plus a grammar-transport regression case pinning legacy behavior (prefill still sent, reasoning still extracted).
  • npm run lint (tsc --noEmit) clean.
  • Full suite: 6436 passed / 1 failed — the one failure (src/sidecar/send-message-concurrency.test.ts) fails identically on pristine origin/main (verified in a separate worktree) and is unrelated.

Fixes #283

Валерий Брижатюк and others added 2 commits August 31, 2026 15:46
…roviders

The trailing `<think>` prefill (and Gemma turn-framing tokens) are
llama-server text-completion artifacts: the local template expects the
open tag pre-typed at the generation point. On the native-tools chat
transport the same prompt ships as a chat message to an OpenAI-
compatible endpoint, where the literal tag is at best noise the model
echoes back — and at worst corrupted server-side: Ollama Cloud mangles
literal `<think>`/`</think>` strings in message content
(ollama/ollama#17248), the trigger for #283.

The injection only fired in hybrid configs (a local llama-server
probing a think-tag model while completions route to a cloud
provider), but there it also mis-parsed clean cloud replies:
`normalizeContent` re-prepended the open tag and the stream parser
started pre-opened, so a reply that never emitted `</think>` was
swallowed whole as reasoning.

- build-prompt: new `suppressReasoningPrefill` input drops the
  trailing reasoning prefill, the Gemma turn framing, and the
  reasoning system token; the step executor sets it for
  `toolTransport: "native_tools"`.
- step-executor: parsing no longer assumes a prefill that was not sent
  (`promptCarriesReasoningPrefill` / `completionAssumesOpenReasoning`);
  grammar-parsed completions keep the legacy prepend — the GBNF
  prelude root structurally starts mid-think — including on
  cross-transport fallover. The one-shot repair prompt stops
  re-appending `<think>`.
- profile-invariants: `checkProfilePromptAligned` learns the
  suppressed shape (the prompt must NOT end with a reasoning prelude).
- provider-presets: document the upstream Ollama Cloud corruption next
  to the preset.

The issue's proposed blanket rewrite of thinking-tag strings in all
outgoing content for ollama.com endpoints is deliberately NOT
implemented: silently mutating user text and tool results is the same
silent-corruption class relocated client-side, and the server-side
half is Ollama's bug to fix.

Fixes #283

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-transport fallover

Adversarial review of the prefill-suppression fix (#283) found a real
streaming regression in the documented default hybrid chain —
cloud (native_tools) primary with a grammar local last resort
(`appendLocal`) on a think-tag profile:

- consumeStream keyed `preOpenedThink` off the PRIMARY transport, but a
  grammar-served fallover stream starts mid-`<think>` (the GBNF prelude
  root emits `body "</think>"` with no open tag), so the parser
  silently swallowed the reasoning text — no live `reasoning_delta`s
  for every sticky-override turn of an outage, where main surfaced
  them. The `servedTransport` stamp on the stream's return value
  arrives only after the last delta, too late to reconfigure a parser.

Fixes, in dependency order:

- completion-types: `StreamChunk.servedTransport` — the fallback
  streamer seam now stamps the serving link's transport on EVERY chunk,
  not just the final result, so live consumers can adapt up front.
- step-executor.consumeStream: the stream parser is created lazily off
  the first chunk's stamp (primary transport when unstamped, i.e. the
  direct non-fallback path), restoring live reasoning classification
  for grammar-served fallover streams.
- llm-fallback-seam: per-link prompt substitution. The main prompt for
  a native-tools primary is prefill-suppressed, which handed the
  grammar fallover link a prompt/template mismatch (GBNF still forces
  mid-think output). `LlmStreamParams.grammarPrompt` carries a lazy,
  memoized prefill-carrying variant built only when a grammar link is
  actually chosen; the one-shot repair retry rebuilds both variants
  repair-shaped so a fallover retry never sees the stale base prompt.
- completionAssumesOpenReasoning now keys purely off the served/parse
  transport: grammar-served output always continues an open think
  block; a chat completion never does. This also stops the (documented-
  unsupported) grammar-primary -> native-link ordering from swallowing
  a clean chat reply whole as reasoning; the literal prefill still
  shipping to the chat link in that reverse ordering remains, matching
  AGENTS.md's "order native-tools links at or above the first
  grammar-only link".
- profile-invariants: the prefill-suppressed branch also flags leaked
  Gemma turn-framing tails, not just reasoning open tags.

Coverage for the pinned invariant 8 gap that let this slip past CI:
fallback-e2e now exercises think-tag profiles through the REAL seam
factories, unary and streaming (live deltas + per-link prompt shape),
llm-fallback-seam.test pins the per-chunk stamp and the grammarPrompt
substitution, and step-executor.test pins both fallover directions.
All 8 new tests fail without the src changes (verified by stashing).

lint clean; agent+prompt+llm 877/877; runtime+tui/providers 336/336;
full suite 6447/6448 — the one failure
(sidecar/send-message-concurrency) fails identically on pristine
origin/main in a clean worktree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@plombeer31
plombeer31 merged commit f393e42 into main Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant