[Experimental] Chat generative-UI webview inset + transport (inline A2UI primitives)#323766
Open
MichaelDanCurtis wants to merge 13 commits into
Open
[Experimental] Chat generative-UI webview inset + transport (inline A2UI primitives)#323766MichaelDanCurtis wants to merge 13 commits into
MichaelDanCurtis wants to merge 13 commits into
Conversation
Author
|
Companion (extension half of this feature): MichaelDanCurtis/vscode-copilot-chat#1 — note |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces experimental primitives in the workbench chat renderer to host an interactive webview “generative UI” inset inside a chat response bubble and to transport messages to/from that inset via internal commands. It threads a new progress/content-part kind (generativeUIRuntimeInset) through the ext host DTO → main thread → chat model/view model → renderer pipeline, and adds a registry to replay state after list virtualization remounts.
Changes:
- Adds a new chat content part (
ChatGenerativeUIInsetPart) that mounts anIWebviewElementinline and relays READY/INTERACTION/RESIZE messaging. - Adds a module-level inset registry plus internal command
_a2ui.postToSurfaceto route host→inset messages and to persist/replay the latest RENDER + STATE_DELTAs. - Extends the proposed API surface and progress DTO plumbing to carry
surfaceId,runtimeUri,initialDoc, andversion.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/vscode-dts/vscode.proposed.chatParticipantAdditions.d.ts | Adds the proposed generativeUI(...) stream method signature. |
| src/vs/workbench/contrib/chat/common/model/chatViewModel.ts | Extends renderer content union to include the new inset part. |
| src/vs/workbench/contrib/chat/common/model/chatSessionOperationLog.ts | Updates equality handling for the new progress kind. |
| src/vs/workbench/contrib/chat/common/model/chatModel.ts | Adds IChatGenerativeUIInset to the progress content model. |
| src/vs/workbench/contrib/chat/common/chatService/chatService.ts | Extends chat progress union to include the inset kind. |
| src/vs/workbench/contrib/chat/browser/widget/chatListRenderer.ts | Wires rendering of generativeUIRuntimeInset to the new content part. |
| src/vs/workbench/contrib/chat/browser/widget/chatContentParts/chatGenerativeUIInsetRegistry.ts | New registry + _a2ui.postToSurface command; records and replays inset state. |
| src/vs/workbench/contrib/chat/browser/widget/chatContentParts/chatGenerativeUIInsetPart.ts | New inline webview inset part with message relay, CSP HTML, intrinsic sizing, and remount logic. |
| src/vs/workbench/api/common/extHostChatAgents2.ts | Implements generativeUI(...) on the ext host response stream (emits progress DTO). |
| src/vs/workbench/api/common/extHost.protocol.ts | Adds IChatGenerativeUIInsetDto to the ext host protocol unions. |
| src/vs/workbench/api/browser/mainThreadChatAgents2.ts | Revives runtimeUri from UriComponents for the inset model. |
Make the inset actually load the bundled runtime: add the runtime's containing directory to the webview localResourceRoots and rewrite the file: URI via asWebviewUri so the <script src> resolves through the webview resource authority. CSP script-src uses webviewGenericCspSource; strict CSP otherwise (no remote scripts, no inline handlers).
Register each ChatGenerativeUIInsetPart by surfaceId in a module-level registry (mirrors the _chat.resizeImage pattern) and deregister on dispose (guarded last-wins). Add internal command _a2ui.postToSurface that looks up the inset by surfaceId and forwards a HostToInset-shaped message to its webview. Dumb lookup+forward pipe; no-op if the inset is gone. This is the host->inset half of the live STATE_DELTA transport.
… via _a2ui.routeInteraction
…store (blank-after-scroll)
…t row recycling (fixes blank-after-scroll)
… bound replay deltas, validate inset messages, align initialDoc type
MichaelDanCurtis
force-pushed
the
feature/inline-a2ui-mcp
branch
from
7f4bd31 to
71d36cf
Compare
MichaelDanCurtis
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Primitives that let a chat participant render an interactive webview surface inside a chat message bubble and exchange messages with it — used by a companion extension PR to render agent-authored "A2UI" documents and stream live MCP data.
All A2UI/MCP semantics live in the extension; core stays a dumb, reusable transport.
Try it / where to start (developer kit)
Everything you need to understand and use this lives in the runtime repo — MichaelDanCurtis/a2ui-runtime:
README.md+docs/(document format, every component + props, the binding model, the four capabilities, extending the catalog, and the security model).doc(card({ title: 'Sales' }, chart({ kind: 'area', bind: 'series' }))), instead of hand-writing JSON.npm install && npm run build, then opengallery/index.htmlto see every component plus the interaction round-trip, live data, and panel/multi-view features rendering (no VS Code needed).To exercise the whole feature end-to-end inside VS Code, pair this PR with the companion extension PR (linked below).
Changes (core)
ChatGenerativeUIInsetPart(chatContentParts/chatGenerativeUIInsetPart.ts) — a chat content part that hosts anIWebviewElementinside a bubble: loads an extension-provided runtime asset, relays messages in/out, tracks intrinsic height, applies a strict CSP (default-src 'none'; script only from the webview source;img-src https: data:). Adapted from the existingChatMcpAppModelwebview-inset pattern.chatGenerativeUIInsetRegistry.ts) —_a2ui.postToSurface(host → inset, by surfaceId) and the extension-registered_a2ui.routeInteraction(inset → extension). Internal, underscore-prefixed.generativeUIonchatParticipantAdditions— rides the existing enabled proposal rather than inventing a new one; the chat model carries agenerativeUIRuntimeInsetcontent part through the ext-host DTO → main-thread → view-model → renderer chain.RENDER+ orderedSTATE_DELTAs per surface so the inset survives the chat list disposing/recreating its row on scroll (replays full state on each webviewREADY; stable per-surface webview origin). Fixes a blank-after-scroll regression.Companion PR
The consumer (the
render_a2uitool, surface manager, MCP pipe, panel host) is in microsoft/vscode-copilot-chat (linked in a comment). The webview runtime this loads — component catalog, typed authoring SDK, and a live examples gallery — lives at MichaelDanCurtis/a2ui-runtime.Notes
npm run typecheck-clientclean for the changed files. Rebased onto currentmain— mergeable, no conflicts. Shared as a working reference for the inset/transport approach; feedback and iteration welcome.