Skip to content

Latest commit

 

History

History

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

streaming-chat

End-to-end demo of devframe's streaming-channel API combined with shared state for persistent chat history. Mirrors the AI-deltas use case from vitejs/devtools#306: the server emits synthesized "tokens" one at a time over a streaming channel, while the conversation log lives in a devframe sharedState so it survives reloads, syncs across panels, and replays cleanly when a client (re)joins mid-stream.

What it shows

  • A scoped context (ctx.scope('example:streaming-chat')) is the preferred entry point - it auto-namespaces every id.
  • my.rpc.streaming.create('tokens', opts) registers a streaming channel (example:streaming-chat:tokens).
  • my.rpc.sharedState('history', …) keeps the message log on the server (example:streaming-chat:history). Each send action appends a user + assistant pair atomically.
  • The producer streams tokens via the channel for low-latency rendering, then commits the joined content back to the shared state when it's done - so refreshes and new clients see the finished message immediately.
  • reader.cancel() aborts mid-stream; the assistant message is marked cancelled: true with whatever content was accumulated.
  • replayWindow: 1024 means a panel reopened mid-stream replays the buffered tokens before resuming live.

Run it

pnpm -C examples/streaming-chat run build
pnpm -C examples/streaming-chat run dev

Then open http://localhost:9897/ - type a prompt, watch tokens stream in, refresh the page mid-conversation, cancel a long answer, click Clear to wipe the log.

Run the tests

pnpm -C examples/streaming-chat run test

Tests boot the server in-process and exercise the full WS round-trip: happy path with shared-state commit, multi-turn history, cancellation with partial content, clear, and replay-after-finish.

Wire it to a real LLM

Replace fakeTokens(prompt) in src/devframe.ts with anything that yields strings - the rest of the example doesn't care. For OpenAI:

const response = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  stream: true,
  messages: [{ role: 'user', content: prompt }],
})
for await (const chunk of response) {
  if (stream.signal.aborted)
    break
  const token = chunk.choices[0]?.delta?.content
  if (token) {
    stream.write(token)
    acc += token
  }
}
stream.close()

stream.signal propagates cancellation from the browser → server → openai.chat.completions.create's own AbortController, so cancelling also stops the upstream request.