Skip to content

fix(dataobj): Avoid deadlock in index-builder when cancelling mid-build - #20621

Merged
benclive merged 3 commits into
mainfrom
benclive/fix-deadlock-in-index-builder
Jan 30, 2026
Merged

fix(dataobj): Avoid deadlock in index-builder when cancelling mid-build#20621
benclive merged 3 commits into
mainfrom
benclive/fix-deadlock-in-index-builder

Conversation

@benclive

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
Fixing a deadlock that can occur in the index-builder if the build context is cancelled mid-build.

  • I believe this occurs on line 382 (of the updated file), when the build process exits on cancelled context before reading all values from the downloadedObject channel.
  • There is a skipMode mechanism which is meant to drain the channel of pending downloads if a build exits, but it didn't trigger in this case. I decided to replace it with different semantics for the downloadWorker and a defer: Every request receives a response, and the download worker only exits when it's input channel is closed.
  • I wasn't able to write a test for this case: I'm open to suggestions for how to do it
@benclive
benclive requested a review from a team as a code owner January 30, 2026 12:34
case obj = <-si.downloadedObjects:
downloadRequestsPending--
case <-ctx.Done():
return "", processed, ctx.Err()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this early return is the cause. Returning here will stop reading from the downloadedObjects channel, despite us writing len(events) to it in the previous for-loop.
Nothing else reads from the downloadedObjects channel, and it has a small buffer to avoid having too many objects in memory at any one time. The next build could fill the input downloadQueue with events, block, and never be able to proceed to reading from this channel.

@benclive
benclive merged commit 294d574 into main Jan 30, 2026
88 checks passed
@benclive
benclive deleted the benclive/fix-deadlock-in-index-builder branch January 30, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 participants