Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix querier panics #4790

Merged
merged 8 commits into from
Mar 5, 2025
Merged

Fix querier panics #4790

merged 8 commits into from
Mar 5, 2025

Conversation

joe-elliott
Copy link
Member

@joe-elliott joe-elliott commented Mar 4, 2025

What this PR does:
A panic was introduced here by abusing the ring forEach function to collect results. replicationSet.Do can return before all reads are complete if quorum is reached. Under certain, rare circumstances this caused the forEach function to modify a result here after it has already been returned and was being marshalled by the gRPC stack into proto.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott merged commit 6d70705 into grafana:main Mar 5, 2025
15 checks passed
@joe-elliott joe-elliott added type/bug Something isn't working backport release-v2.7 Backport for V2.7 labels Mar 25, 2025
Copy link
Contributor

The backport to release-v2.7 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-4790-to-release-v2.7 origin/release-v2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 6d707056521ce46c79179fc16fc12126579f66a0

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-4790-to-release-v2.7
# Create the PR body template
PR_BODY=$(gh pr view 4790 --json body --template 'Backport 6d707056521ce46c79179fc16fc12126579f66a0 from #4790{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title '[release-v2.7] Fix querier panics' --body-file - --label 'type/bug' --label 'backport' --base release-v2.7 --milestone release-v2.7 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-4790-to-release-v2.7

# Create a pull request where the `base` branch is `release-v2.7` and the `compare`/`head` branch is `backport-4790-to-release-v2.7`.

# Remove the local backport branch
git switch main
git branch -D backport-4790-to-release-v2.7
joe-elliott added a commit that referenced this pull request Mar 25, 2025
* fix forIngesterRings

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix generators

Signed-off-by: Joe Elliott <number101010@gmail.com>

* refactor to pass the funcs in directly

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix breaks

Signed-off-by: Joe Elliott <number101010@gmail.com>

* improve error

Signed-off-by: Joe Elliott <number101010@gmail.com>

* got rid of metrics collector

Signed-off-by: Joe Elliott <number101010@gmail.com>

* chagnelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix found behavior

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
(cherry picked from commit 6d70705)
joe-elliott added a commit that referenced this pull request Mar 25, 2025
* fix forIngesterRings

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix generators

Signed-off-by: Joe Elliott <number101010@gmail.com>

* refactor to pass the funcs in directly

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix breaks

Signed-off-by: Joe Elliott <number101010@gmail.com>

* improve error

Signed-off-by: Joe Elliott <number101010@gmail.com>

* got rid of metrics collector

Signed-off-by: Joe Elliott <number101010@gmail.com>

* chagnelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix found behavior

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
(cherry picked from commit 6d70705)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport release-v2.7 Backport for V2.7 backport-failed type/bug Something isn't working
2 participants