[backend-scheduler] fix race condition between scheduler and compaction provider #5409
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zalegrala
force-pushed
the
compactionProviderSkipRecentBlocks
branch
from
ca09212 to
d64ab83
Compare
3 tasks
mdisibio
reviewed
Jul 16, 2025
mdisibio
reviewed
Jul 17, 2025
zalegrala
force-pushed
the
compactionProviderSkipRecentBlocks
branch
from
8f33b52 to
1173b8d
Compare
zalegrala
requested review from
electron0zero,
ie-pham,
javiermolinar,
joe-elliott,
mapno,
stoewer and
yvrhdn
as code owners
zalegrala
requested review from
carles-grafana and
ruslan-mikhailov
as code owners
mdisibio
approved these changes
Jul 18, 2025
knylander-grafana
pushed a commit
to knylander-grafana/tempo-doc-work
that referenced
this pull request
Jul 18, 2025
…on provider (grafana#5409) * [backend-scheduler] fix race condition between scheduler and compaction provider * Protect the recentJobs map with a mutex * Renames for clarity * Convert map values to UUIDs for less conversion * Validate the prune age is twice the compaction cycle * Validate the measurement interval is sufficiently fast * Fix test for UUID change and config validation * Update integration test for change in behavior * Reduce channel size by removing duplciate initialization * Simplify append loop * Give a little more time * Reword the validation error * Record the job before its sent * Update integration test notes * Update changelog * Update config after rebase * Update test config after rebase
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
This is an alternative approach to the same issue described in #5405.
Here we fix the race by keeping track of the recent jobs which were created and their blocks. This cache is used to skip blocks on the block selector from being considered. Once the next blockselector is called, we use the job list to identify what cache needs to be pruned. This allows a small delay between when the compaction provider created the job and when the scheduler assigns the job, but without the additional overhead of looping through the work list as proposed in #5405.
Which issue(s) this PR fixes:
Fixes #5405
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]