Skip to content

[Bugfix] Deadlock on invalid query to api/v2/search/tags - #5607

Merged
ruslan-mikhailov merged 6 commits into
grafana:mainfrom
ruslan-mikhailov:bugfix/deadlock
Sep 4, 2025
Merged

[Bugfix] Deadlock on invalid query to api/v2/search/tags#5607
ruslan-mikhailov merged 6 commits into
grafana:mainfrom
ruslan-mikhailov:bugfix/deadlock

Conversation

@ruslan-mikhailov

@ruslan-mikhailov ruslan-mikhailov commented Sep 4, 2025

Copy link
Copy Markdown
Contributor

What this PR does

It fixes deadlock in ingesters.

  1. panic in ExecuteTagNames on nil pointer dereference, when rootExpr is nil, which can happen when query is invalid
  2. after recovering from panic, mutex unlock is not defered, which results in deadlock.

Impact: deadlock for read and write until ingester's restart for tenant who triggered panicked request to api/v2/search/tags

How to reproduce

  1. send request to api/v2/search/tags with invalid query. Example:
--> GET http://localhost:3100/api/v2/search/tags
--> query params: {'q': '{ a="test" }'}
<-- status: 500
<-- body: error querying ingesters in Querier.SearchTags: failed to execute f() for x.x.x.x:x: rpc error: code = Unknown desc = recovered in SearchTagsV2
  1. Try to search by trace id, it will always return 500. Example:
--> GET http://localhost:3100/api/traces/19a8654f9aaa456b7e45d19ea770ac03
<-- status: 500
<-- body: error querying ingesters in Querier.FindTraceByID: context deadline exceeded

Which issue(s) this PR fixes

Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
@ruslan-mikhailov ruslan-mikhailov changed the title [Bugfix] Sep 4, 2025
Comment thread modules/ingester/instance_search.go Outdated
Comment thread pkg/traceql/engine_test.go
@ruslan-mikhailov
ruslan-mikhailov merged commit a6cd425 into grafana:main Sep 4, 2025
37 of 38 checks passed
@ruslan-mikhailov
ruslan-mikhailov deleted the bugfix/deadlock branch September 4, 2025 11:46
ruslan-mikhailov added a commit to ruslan-mikhailov/tempo that referenced this pull request Sep 5, 2025
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ruslan-mikhailov added a commit that referenced this pull request Sep 5, 2025
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ie-pham pushed a commit to ie-pham/tempo that referenced this pull request Jan 15, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
@ie-pham ie-pham added type/bug Something isn't working backport release-v2.8 labels Jan 15, 2026
ie-pham pushed a commit to ie-pham/tempo that referenced this pull request Jan 15, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ie-pham added a commit that referenced this pull request Jan 15, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2

Co-authored-by: Ruslan Mikhailov <195758209+ruslan-mikhailov@users.noreply.github.com>
mattdurham pushed a commit to mattdurham/tempo that referenced this pull request Jun 18, 2026
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport release-v2.8 type/bug Something isn't working

3 participants