Skip to content

Conversation

@ruslan-mikhailov
Copy link
Contributor

@ruslan-mikhailov ruslan-mikhailov commented Sep 4, 2025

What this PR does

It fixes deadlock in ingesters.

  1. panic in ExecuteTagNames on nil pointer dereference, when rootExpr is nil, which can happen when query is invalid
  2. after recovering from panic, mutex unlock is not defered, which results in deadlock.

Impact: deadlock for read and write until ingester's restart for tenant who triggered panicked request to api/v2/search/tags

How to reproduce

  1. send request to api/v2/search/tags with invalid query. Example:
--> GET http://localhost:3100/api/v2/search/tags
--> query params: {'q': '{ a="test" }'}
<-- status: 500
<-- body: error querying ingesters in Querier.SearchTags: failed to execute f() for x.x.x.x:x: rpc error: code = Unknown desc = recovered in SearchTagsV2
  1. Try to search by trace id, it will always return 500. Example:
--> GET http://localhost:3100/api/traces/19a8654f9aaa456b7e45d19ea770ac03
<-- status: 500
<-- body: error querying ingesters in Querier.FindTraceByID: context deadline exceeded

Which issue(s) this PR fixes

Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
@ruslan-mikhailov ruslan-mikhailov changed the title [Bugfix] Sep 4, 2025
@ruslan-mikhailov ruslan-mikhailov merged commit a6cd425 into grafana:main Sep 4, 2025
37 of 38 checks passed
@ruslan-mikhailov ruslan-mikhailov deleted the bugfix/deadlock branch September 4, 2025 11:46
ruslan-mikhailov added a commit to ruslan-mikhailov/tempo that referenced this pull request Sep 5, 2025
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
ruslan-mikhailov added a commit that referenced this pull request Sep 5, 2025
+ [Bugfix] Fix potential deadlock in SearchTagValuesV2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants