Fix panic in query-frontend when combining results by mapno · Pull Request #3683 · grafana/tempo

mapno · 2024-05-16T08:01:28Z

What this PR does:

Fixes a panic when combining results in the query frontend.

Panic stack trace

2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1030 +0x125
2024-05-15 13:01:13.396	created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 1386
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1019 +0x8b
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).serveStreams.func2.1()
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1784 +0xe3a
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).handleStream(0xc000725600, {0x3161420, 0xc0006455f0}, 0xc046aef680)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1670 +0x11e7
2024-05-15 13:01:13.396	google.golang.org/grpc.(*Server).processStreamingRPC(0xc000725600, {0x3144358, 0xc046afebd0}, {0x3161420, 0xc0006455f0}, 0xc046aef680, 0xc04193e780, 0x484a9e0, 0x0)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1506 +0x85
2024-05-15 13:01:13.396	google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046ab0c60?)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:77 +0xab
2024-05-15 13:01:13.396	github.com/grafana/dskit/middleware.GRPCServerLog.StreamServerInterceptor({{0x31201e0?, 0xc000642b90?}, 0x60?, 0xc2?}, {0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046af0d00)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.396	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0})
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/opentracing-contrib/go-grpc/server.go:114 +0x34a
2024-05-15 13:01:13.396	github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1({0x268c620, 0xc0416ca5b0}, {0x3153a00, 0xc0469ff0e0}, 0xc040e11f50, 0xc046af0d40)
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.396	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000})
2024-05-15 13:01:13.396		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:44 +0xa4
2024-05-15 13:01:13.395	github.com/grafana/dskit/middleware.StreamServerInstrumentInterceptor.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0xc040e11f50, 0xc046af0f00)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/vendor/google.golang.org/grpc/server.go:1515 +0xb2
2024-05-15 13:01:13.395	google.golang.org/grpc.getChainStreamHandler.func1({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/cmd/tempo/app/app.go:156 +0x9c
2024-05-15 13:01:13.395	github.com/grafana/tempo/cmd/tempo/app.(*App).setupAuthMiddleware.func2({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0xc040e11f50, 0x2a51190)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/vendor/github.com/grafana/dskit/middleware/grpc_auth.go:53 +0xb2
2024-05-15 13:01:13.395	github.com/grafana/dskit/middleware.StreamServerUserHeaderInterceptor({0x268c620, 0xc0416ca5b0}, {0x3154030, 0xc0474ae000}, 0x7e9dc30c8ab0?, 0x2a51190)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/tempopb/tempo.pb.go:4100 +0x107
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/tempopb._StreamingQuerier_Search_Handler({0x268c620, 0xc0416ca5b0}, {0x3153898, 0xc0474ae020})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/frontend.go:171 +0x2d
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend.(*QueryFrontend).Search(0x481c1c0?, 0x268c620?, {0x3157330?, 0xc046ab0db0?})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/search_handlers.go:60 +0x8f1
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend.newSearchStreamingGRPCHandler.func1(0xc046af0f40, {0x3157330, 0xc046ab0db0})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/pipeline/collector_grpc.go:53 +0x207
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/pipeline.GRPCCollector[...].RoundTrip(0x20?, 0xc046aefd40)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/combiner/common.go:77 +0x357
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/combiner.(*genericCombiner[...]).AddResponse(0x31701c0, {0x31318d8, 0xc047167230})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/modules/frontend/combiner/search.go:31 +0xd4
2024-05-15 13:01:13.395	github.com/grafana/tempo/modules/frontend/combiner.NewSearch.func2(0xc04194b2a0, 0xc0474ae060, {0x3139c60?, 0xc04194b2a0?})
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/traceql/combine.go:24 +0x95
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/traceql.(*MetadataCombiner).AddMetadata(0xc00049f9a0, 0xc0005ac230)
2024-05-15 13:01:13.395		/Users/mapno/workspace/github.com/grafana/tempo/pkg/traceql/combine.go:84 +0x1e5
2024-05-15 13:01:13.395	github.com/grafana/tempo/pkg/traceql.combineSearchResults(0xc000564460, 0xc0005ac230)
2024-05-15 13:01:13.395	goroutine 1390 [running]:
2024-05-15 13:01:13.395	
2024-05-15 13:01:13.395	panic: assignment to entry in nil map

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

electron0zero

tested, it works

* Fix panic in query-frontend when combining results * chlog

Fix panic in query-frontend when combining results

0bf1860

mapno requested review from annanay25, electron0zero, ie-pham, joe-elliott, mdisibio, stoewer, yvrhdn and zalegrala as code owners May 16, 2024 08:01

chlog

635c1d9

mapno added type/bug Something isn't working area/query component/query-frontend size/S and removed area/query size/S labels May 16, 2024

electron0zero approved these changes May 16, 2024

View reviewed changes

mapno merged commit 805b3f5 into grafana:main May 16, 2024

mapno deleted the combine-panic branch May 16, 2024 13:23

mattdurham pushed a commit to mattdurham/tempo that referenced this pull request Jun 18, 2026

Fix panic in query-frontend when combining results (grafana#3683)

b3ce555

* Fix panic in query-frontend when combining results * chlog

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix panic in query-frontend when combining results - #3683

Fix panic in query-frontend when combining results#3683
mapno merged 2 commits into
grafana:mainfrom
mapno:combine-panic

mapno commented May 16, 2024

electron0zero left a comment

Labels

2 participants

Uh oh!

Conversation

mapno commented May 16, 2024

electron0zero left a comment

Choose a reason for hiding this comment

Labels

2 participants