fix(agent): never leak raw reasoning as a reply; transport-aware prompt - #286
Merged
Merged
Conversation
Two-part fix for issue #285 (verbatim chain-of-thought delivered to the user as the agent's reply, and native_tools having no effect on model behaviour). Guardrail (step-executor): the native_tools reasoning-only salvage no longer wraps an unparseable `reasoning_content` body into a `reply { text }` call. A GBNF-shaped batch embedded in the think channel is still recovered as real tool calls, but anything else now returns a parse failure and routes through the existing one-shot repair (parse_retry + buildToolCallRepairPrompt, REPAIR_MAX_TOKENS-capped). A repair that fails too ends the step as a GrammarError — a parse error, not a CoT leak. Root cause (prompt): the stable prefix was transport-blind — under native_tools the request carried an OpenAI `tools` array with `tool_choice: "auto"` while the prompt simultaneously ordered "Emit a JSON ARRAY of tool calls now" and the persona mandated "exactly one JSON array". `buildStablePrefix` now takes `toolTransport` (threaded from `StepDependencies` through `BuildPromptInput`): under native_tools the persona's emission mandate and the `### instructions` block switch to native function-calling guidance. The `### tools` text catalog stays in both modes — the provider fallback chain can hand a native-shaped request to a grammar-only llama-server link, and the catalog carries the tier / `tool.view` semantics. The grammar-path prefix is byte-identical to before (KV-cache safe; verified by sha256 against the previous implementation across profile/turn-framing/win32/persona variants). Known caveat: on a native-to-grammar mid-session fallover the prompt lacks the JSON-array mandate while the grammar link parses GBNF. This is acceptable because llama-server's GBNF grammar constrains decoding to the array shape regardless of the prompt mandate. Fixes #285 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eplay drift Adversarial review of the issue #285 fix found three residual problems; this commit closes all three. 1. `### rules` still opened with the text-array mandate ("One tool-call array per step ... a solo action is a length-1 array") in BOTH transports, contradicting the native `### instructions` block in the same prefix. The rules line is now transport-variant like the persona and instructions: the grammar line is byte-identical to before (sha256-verified against v0.4.2 across omitted/explicit, win32, persona-override, reasoning-token and turn-framing variants), the native line mandates the function-calling interface. The same sweep caught one more survivor in the shared persona — "if another tool is next, emit that tool JSON" — which likewise becomes "call that tool" under native_tools only. 2. `buildToolCallRepairPrompt` was transport-blind: on the exact parse_retry path the #285 fix routes reasoning-only completions through, it appended "Emit a corrected JSON array only" / "Use a length-1 array" onto a native prefix that forbids text-JSON — the dual mandate recreated at the one retry a failing model gets. The repair mandate now follows `deps.toolTransport`: native repairs order a corrected native tool call, grammar repairs keep the legacy lines byte-for-byte. A new probe test captures the second llmComplete prompt under native transport and asserts no text-array mandate survives anywhere in it. 3. `atomic-agent trace replay` regressed into 100% false drift for native-transport sessions: the prefix now differs by construction per transport, but `replaySession` always rebuilt the grammar variant and traces do not record which transport served the session. The replay is now transport-aware: `ReplayContext.toolTransport` pins the comparison when the caller knows the transport; when omitted (the trace-command case) both variants are built and a recorded hash is clean when it matches either — old (pre-#285) traces keep matching through the byte-identical grammar variant. Each step reports which variant matched (`matchedTransport`). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two-part fix for the reasoning-channel leak and the
native_toolsprompt contradiction reported in #285.1. Guardrail — stop salvaging raw CoT as a reply (
src/agent/step-executor.ts)tryParseToolCalls'snative_toolsbranch handled a reasoning-only completion (emptycontent, notool_calls, non-emptyreasoning_content) by wrapping the raw reasoning text into areply { text }call — delivering the model's internal chain-of-thought verbatim as deliberate agent speech, exactly as the reporter's transcript shows.Now:
[{tool, args}, ...]batch embedded in the think channel is still recovered as real tool calls (that path recovers genuine intent and is kept);{ ok: false }, routing the completion through the existing one-shot repair path (parse_retryevent +buildToolCallRepairPrompt+REPAIR_MAX_TOKENScap) that already handles every other unparseable body;GrammarError— a normal parse error, never a CoT leak.2. Root cause — transport-aware stable prefix (
src/prompt/stable-prefix.ts+ threading)Under
llm.toolTransport: "native_tools"the request carries an OpenAItoolsarray withtool_choice: "auto", while the prompt simultaneously ordered "Emit a JSON ARRAY of tool calls now" and the persona mandated "exactly one JSON array". The reporter measured 0/6 completions using nativetool_callsunder this dual mandate.buildStablePrefixnow takestoolTransport(threadedStepDependencies→BuildPromptInput→buildStablePrefix): undernative_toolsthe persona's emission mandate and the### instructionsblock switch to native function-calling guidance ("call tools through the function-calling interface;reply/ plain text for the final answer").The
### toolstext catalog is deliberately kept in both modes: the provider fallback chain can hand a native-shaped request to a grammar-only llama-server link (see thebuildLlmStreamParamscomment about keepinggrammarpopulated), and the catalog carries the tier /tool.viewsemantics. The JSON-array emission mandate was the contradiction, not the catalog.Byte-stability
The grammar-path prefix is byte-identical to
origin/main's output — verified by sha256-comparingbuildStablePrefixfrom both revisions across variants (plain,reasoningSystemToken, Gemma-4turnSystemOpen,maxParallelToolCalls, win32, customsystemPersona): all identical (fde2b2b9…). KV-cache reuse is unaffected for local models; within a session the transport is fixed, so the native prefix is byte-stable across turns too (pinned by a test).Known caveat
On a native-to-grammar mid-session fallover the prompt lacks the JSON-array mandate while the grammar link parses GBNF. Acceptable: llama-server's GBNF grammar constrains decoding to the array shape regardless of the prompt mandate.
Test evidence
native_tools: unparseable reasoning-only completion routes through parse_retry, never leaks CoT as a reply— asserts the repair round-trip and that no reply anywhere in the transcript carries the reasoning body.native_tools: recovers a GBNF-shaped batch embedded in reasoning_content without a retry(salvage of real tool calls is preserved).native_tools: reasoning-only on both attempts surfaces a parse error, not the CoT(rejects withGrammarError).buildPrompt tool transport (issue #285)block: native prefix omits both text-JSON mandates but keeps### tools; grammar prefix byte-identical whether the field is omitted or explicit; native prefix byte-stable across turns; explicitsystemPersonaoverride wins on both transports.npm run lintclean. Full suite: 610/611 files, 6436 passed / 1 failed — the one failure (src/sidecar/send-message-concurrency.test.ts) fails identically on pristineorigin/main@ 4caff55 and is unrelated to this diff.Fixes #285