MCP image content blocks can never reach the model: executeCode results are text-only #327

Summary

MCP tool results can carry {type: "image"} content blocks, and the MCP gatekeepers faithfully deliver them to the agent's bindings — but the agent can never actually see them. The only route from a binding back into the model context is the executeCode tool, whose result is text-only, so every image block dies at that boundary. Any MCP server that answers with images (screenshots, charts, rendered pages, camera stills, PDF page renders) is invisible to the agent by construction.

This looks like a dangling end between two halves of the codebase that each already support images.

The path, as it is today

1. packages/mcp-shared/src/tools.ts — toCallResult() preserves the full content-block array of a tools/call result, including image blocks. So env.MY_MCP.someTool() inside executeCode really does receive the image (base64 + mimeType).
2. packages/workshop-backend/src/overseer.ts — executeCodeMode() builds the tool result from the collected console log alone (the trace items joined into a string). The executed module's return value is discarded by CODE_MODE_HARNESS (await agent(self, env, this.ctx); — nothing is captured).
3. packages/workshop-backend/src/agent.ts — the executeCode tool wraps that string as content: [{type: "text", ...}]. Every other tool does the same.

So the best an agent can do is console.log the base64, which (a) is not vision — the model sees characters, not pixels; (b) is token-expensive; and (c) rides the whole result through storage as text.

The pieces to fix it already exist

• The model layer is already multimodal including tool results: pi-ai's ToolResultMessage.content is (TextContent | ImageContent)[], and the providers translate image parts.
• Chat attachments already reach the model as ImageContent (agent.ts).
• agent-compaction.ts already knows how to prune images from projections (replacing them with [image <mime>] markers), so bounding image bytes in long contexts has a natural hook.

The gap is only the plumbing between executeCode and the ToolResultMessage.

Proposed mechanism (sketch)

1. CODE_MODE_HARNESS captures the default function's return value and hands it back from run() (today it is discarded).
2. executeCodeMode() returns {log, images}: images extracted from the returned value when it contains MCP-style {type: "image", data, mimeType} blocks (or a small documented shape), with caps on count and total bytes.
3. agent.ts emits them as ImageContent parts in the tool result, after the log text.
4. Persistence: store only a short marker (e.g. [image image/jpeg, 48 KB]) in the stored tool output. Replayed history then shows the marker, mirroring what compaction already does to old images — Durable Object rows stay small, and images naturally live only within the run that produced them, which is exactly the window where an agent acts on what it saw.

With that in place, the docs could tell agents: "to look at an image a binding returned, return its content blocks from executeCode".

Why it matters

Image-producing MCP servers are common (browser/screenshot servers, chart renderers, vision pipelines), and connecting them is this platform's headline feature — today they all silently degrade to text. This also unblocks screenshot-driven flows (a model observing a GUI through an MCP bridge) without any special-casing: it is the same generic mechanism.

Happy to contribute a PR if the direction sounds right.