Convert backing storage to git, change sync to OT, editor widget to CodeMirror - #275
Merged
Conversation
My .wrangler is actually a symlink, so wasn't being ignored.
My prompt: > Currently, all gadget code in a workspace is stored in a Yjs doc. I would live to move to a model where most code is stored as git commits instead, with Yjs only used to track the uncommitted changes within the context of a particular chat thread. > > To this end I am thinking that each workspace could contain a git object store tracking files and commits. At the workspace level, each gadget is backed by some commit ID. > > I think we only want the object layer of git here. We don't really want or need the ref layer. We are implementing our own workflow and only using git for storage and history. Our "refs" are the gadget records, which we manage in our own way. We don't need git's concept of branches, tags, etc. > > Moreover, if we focus on just the object layer, then I think we can freely throw unrelated git histories into the same storage, rather than having to maintain a separate repository for each gadget. If the gadgets have related histories then we get nice de-duplication; if they don't, whatever. Fable asked several questions before writing the plan.
* Adds new dependencies. * Adds a server-side utility library, git-store.ts, providing git operations. * Adds the appropriate table to the overseer schema. * Adds yjs-seed.ts to the shared code, which will be used by both client and server. (The original plan called for this to be part of git-store.ts, but it was noted that the client needs this too, so it was split out and the plan updated.)
The server code is not implemented yet; new methods throw. The UI code is temporarily rendered out-of-service until it can be updated in a later commit.
This updates the server to store mainline gadget code in git rather than Yjs. Migration from the old format is not implemented yet.
This commit implements the migration of existing gadgets to the new git storage. The code change history is replayed and turned into a commit history, with every past merge point becoming a commit. (Changes made outside of any chat, via a human typing, are batched heuristically -- but nobody actually does that.)
Update the UI, per the plan.
In part 1, we implemented the whole plan, but we ended up with a design choice that seems sub-optimal: the entire state of all gadget code is pinned at the time a chat begins, even if the gadget code is not actually touched within the chat. This can cause a chat to fall behind the mainline unnecessarily. Part 2 of the plan seeks to fix this by only pinning a gadget's code the first time it written within a chat, and unpinning when changes are later merged. (Full details in the plan doc.)
This commit moves the bodies of updateCode(), updateChatFromMainline(), mergeChanges(), revertChanges(), and discardChatDDraftChanges() from OverseerClientInterface to OverseerImpl. The ONLY change made to the bodies is changing `this.impl` to `this`. This is in preparation for work in the next commit. NOTE: `git diff` displays this commit very poorly for some reason; it gets totally de-synced. You'll have to trust me that it's purely cut-and-paste!
These are the backend changes to enable pinning gadgets lazily within a chat, per the plan.
With the recent changes, Yjs has become a source of complexity rather than a solution to it. Operational Transforms would be a much better fit. We might as well switch now so that we don't have to do multiple migrations later. We adopt CodeMirror's OT implementation, and opportunistically go ahead and adopt CodeMirror itself, replacing Monaco, since otherwise we'd have to write an adapter. CodeMirror is, in retrospect, a better fit for us than Monaco anyway!
This had to change significantly since CodeMirror's diff support is different from Monaco's. Moreover, we had built significant customization on top of Monaco which now had to be built a different way.
…roposed changes. If there are proposed changes, the reload is necessary, but if not, we shouldn't reload.
We no longer use Yjs (except to migrate existing data) nor Monaco. We now use isomorphic-git and CodeMirror.
The retry logic landed separately concurrently with most of this branch being developed. Claude caught the discrepency when reviewing the rebase, and recommended this commit to bring the new code in line with everything else.
github-actions
Bot
added
workshop/frontend
Preview:
|
Member
Author
|
Bugs reported by testers:
|
capnweb-validate already validates all this stuff.
This is a more natural name, especially for those not steeped in operational transform literature. CodeOp -> CodeChange, FileOp -> FileChange, TextOp -> TextChange.
The existing ChatGadgetPin, which was an extension of this type, becomes explicitly defined as such, and named ChatGadgetPinState.
Apparently, a previous agent had incorrectly concluded that AiChatSubscriber.changeApplied() messages (delivered at turn end) were sufficient for the "streaming preview", and so it deleted the streaming mechanism. Concerning. This commit brings it back. Fable decided to take a different approach, though. Previously the provisional streaming changes were delivered as Yjs updates, so under the new model you might expect them to be OTs. But Fable decided it's simpler to just stream the edit instruction to the client and let the client apply it, because this way the server doesn't have to materialize the file in order to search for the replacement string in order to generate OTs with correct offsets. The client already has the file materialized. So this is more efficient and probably simpler in the end.
Now it uses `presentableDiff()` instead of char-minimal `diff()`. `presentableDiff()` was intended specifically for this purpose. (This means the highlighting now does a word diff instead of a char diff.)
…anges. This was meant to signal only human changes.
Previously, accepting the very first change in a workspace would drop you out of the current chat and back to the chat selector. This was because the UI very briefly perceived that there were no longer any proposed changes (they'd been accepted), but also there was still no committed code in the gadget (the commit hadn't propagated back to the client yet). So it would swich to simple chat mode, removing the `&chat=0` query param, and then immediately switch back to multi-chat mode, leaving you at the chat selector.
There is a race between a subscription callback and a return. Cap'n Web does not guarantee ordering between a call and a return heading in the same direction, and indeed it can become out-of-order depending on microtask loop details. The problem doesn't manifest when running locally with `wrangler dev`, only in production.
The agent was being told it had to read the file first before editing even in cases where it already knew the content.
kentonv
marked this pull request as ready for review
August 21, 2026 03:55
![devin-ai-integration[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
I think this came from a botched merge conflict resolution. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Maximo-Guk
approved these changes
Aug 21, 2026
ndisidore
approved these changes
Aug 21, 2026
* Extract LegacyWorkspace fixture and add migration content-fidelity oracle. Move the LegacyWorkspace fixture (and its helpers) out of git-migration.test.ts into a shared __tests__/legacy-workspace.ts, parameterized over storage so it can seed either mock storage or a real Durable Object's storage. Add readDocFiles/expectHeadsMatchDoc: an oracle that checks every migrated gadget head against an independent replay of the recorded Yjs update log, applied in three representative unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Test the git-storage migration through the real Overseer constructor trigger. Seed single- and multi-gadget legacy (version-1) workspaces into a real TEST_OVERSEER Durable Object via the LegacyWorkspace fixture, abort all DOs, and let re-entry drive the constructor's blockConcurrencyWhile migration over real SQLite storage. Asserts version 1->2, head trees matching an independent replay of the seeded Yjs log (per gadget), the fallback owner commit identity, and unpolluted per-gadget commit chains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Sync pnpm-lock.yaml with the jose ^6.2.8 specifier from e918adf. The lockfile's specifier entry was left stale when package.json was bumped, so a frozen-lockfile install fails on the mismatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
If, prior to the multi-gadget migration, a workspace had proposed changes only which had not been accepted yet, then the multi-gadget migration (long before this branch) handled them incorrectly, resulting in a workspace which appeared to have no gadgets. The gadget code was abandoned. This change updates the git migration to detect such broken workspaces and repair them. (We never actually heard anyone complain but we might as well repair the issue now that we know about it.)
This message would appear in the chat log at the moment that the workspace was migrated. Moreover, the only way to revert changes that had been pending before migration was to click the revert button on this message. The "discard" button on the "pending changes" banner didn't work, due to a bug. Now this message doesn't appear, and the "discard" button on the banner works as expected.
This was referenced Aug 24, 2026
darjss
pushed a commit
to darjss/cloudflare-os-erxes
that referenced
this pull request
Aug 26, 2026
…odeMirror (cloudflare#275) **LARGE CHANGE ALERT** This change does three things: * Accepted Gadget code is now stored in git format, rather than Yjs. Each workspace has a (hidden) git repository, and each gadget's code is a commit in this repository. (Each gadget is essentially a branch, co-existing within the same repo, even if the change histories are unrelated.) * Pending changes (changes in a chat that haven't been accepted yet) are now represented as operational transforms rather than CRDT. This allows the changes to apply directly on top of the underlying git storage, without the need to copy the contents of git into a CRDT upfront before editing can start. (I had initially tried to continue using Yjs, but the need to "seed" the Yjs doc from git whet starting changes was inefficient and convoluted.) The OT library used is `@codemirror/state`. * Since we switched to CodeMirror's OT library, I looked into CodeMirror in general as an alternative to Monaco, and concluded that it's actually a much better fit for us. So I went ahead and ripped out Monaco in favor of CodeMirror entirely. Otherwise, it would have been necessary to develop a hand-rolled adapter between Monaco and CodeMirror's OT format -- wasted work if we later decided to switch to CodeMirror anyway, which seemed likely. [The plan is documented in `plans/git-storage.md`.](https://github.com/cloudflare/cloudflare-os/blob/main/plans/git-storage.md) Note that the plan is long and composed into three parts. Each part begins with a human-readable overview and then goes into increasing detail intended for an agent to follow. The code was almost entirely written by Fable with Sol acting as code reviewer at each step. I am pretty happy with the code I've reviewed but need to spend a bit more time looking before un-drafting this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LARGE CHANGE ALERT
This change does three things:
@codemirror/state.The plan is documented in
plans/git-storage.md. Note that the plan is long and composed into three parts. Each part begins with a human-readable overview and then goes into increasing detail intended for an agent to follow.The code was almost entirely written by Fable with Sol acting as code reviewer at each step. I am pretty happy with the code I've reviewed but need to spend a bit more time looking before un-drafting this.