fix: Ensure partition-reader starts up correctly #14845
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -103,6 +103,9 @@ func (p *Reader) start(ctx context.Context) error { | |||
|
|
|||
| // We manage our commits manually, so we must fetch the last offset for our consumer group to find out where to read from. | |||
| lastCommittedOffset := p.fetchLastCommittedOffset(ctx) | |||
| if lastCommittedOffset > 0 { | |||
| lastCommittedOffset++ // We want to begin to read from the next offset, but only if we've previously committed an offset. | |||
There was a problem hiding this comment.
I checked Mimir and I can see that they do this too! 👍
There was a problem hiding this comment.
How does this compare with the comment on Line 379:
lastProducedOffset = lastProducedOffset - 1 // Kafka returns the next empty offset so we must subtract 1 to get the oldest written offset.
There was a problem hiding this comment.
If Kafka returns the next empty offset, why do we need to add 1?
There was a problem hiding this comment.
That's the last produced offset, vs last committed offset. Apparently they behave slightly differently!
grobinson-grafana
approved these changes
Nov 8, 2024
This was referenced Dec 23, 2024
This was referenced Feb 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This fixes an edge case where there are no new records available in Kafka during start up, which would cause a deadlock.
It also ensures we don't process the last comitted offset again at startup because we already processed and committed it.
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mddeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR