Describe the bug
When tempo is deployed on k8s via helm chart and k8s sends SIGTERM signal to query-frontend pod due to preemption in order to admit critical pod, query-frontend pod fails to gracefully stop in 30s default window. This means that k8s has to kill it via SIGKILL signal and then this pod will be hanging in error state. As seen in the logs, multiple modules are stopped but it keep logging message below until it is killed
level=warn ts=2025-10-11T00:29:52.146919925Z caller=server.go:1539 method=/frontend.Frontend/Process duration=374.843µs msg=gRPC err="queue is stopped"
Full log
2025-10-11 02:30:22.086
level=warn ts=2025-10-11T00:30:22.086496213Z caller=server.go:1539 method=/frontend.Frontend/Process duration=332.034µs msg=gRPC err="queue is stopped"
2025-10-11 02:30:22.025
level=warn ts=2025-10-11T00:30:22.025431275Z caller=server.go:1539 method=/frontend.Frontend/Process duration=343.21µs msg=gRPC err="queue is stopped"
2025-10-11 02:30:21.884
..... same log multiple times.....
level=warn ts=2025-10-11T00:29:52.165841052Z caller=server.go:1539 method=/frontend.Frontend/Process duration=282.481µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.153
level=warn ts=2025-10-11T00:29:52.153100985Z caller=server.go:1539 method=/frontend.Frontend/Process duration=261.76µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.149
level=warn ts=2025-10-11T00:29:52.149313234Z caller=server.go:1539 method=/frontend.Frontend/Process duration=260.199µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.149
level=warn ts=2025-10-11T00:29:52.149057059Z caller=server.go:1539 method=/frontend.Frontend/Process duration=355.81µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.146
level=warn ts=2025-10-11T00:29:52.14696915Z caller=server.go:1539 method=/frontend.Frontend/Process duration=414.484µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.146
level=warn ts=2025-10-11T00:29:52.146919925Z caller=server.go:1539 method=/frontend.Frontend/Process duration=374.843µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093738737Z caller=module_service.go:120 msg="module stopped" module=overrides
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093691152Z caller=module_service.go:120 msg="module stopped" module=memberlist-kv
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093679631Z caller=module_service.go:120 msg="module stopped" module=overrides-api
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093659119Z caller=module_service.go:120 msg="module stopped" module=query-frontend
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093637758Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.031021272s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093621838Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.031005288s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093590661Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040844602s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093473889Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040857282s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093457553Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040658244s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093443544Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040680388s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093427272Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040643844s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093412296Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.030680343s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093391703Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.030704815s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093245139Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.030534339s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.092
level=info ts=2025-10-11T00:29:52.09272838Z caller=app.go:239 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
To Reproduce
Steps to reproduce the behavior:
- start tempo from helm chart
- send sigterm
Expected behavior
query-fronted should be able to handle termination gracefully and properly exit
Environment:
- Infrastructure: kubernetes
- Deployment tool: helm chart 1.47.1
Additional Context
This issue happened at 2:30 am so there were no queries executed / activity at the time
Perhaps this is also related to #2847
Describe the bug
When tempo is deployed on k8s via helm chart and k8s sends
SIGTERMsignal toquery-frontendpod due to preemption in order to admit critical pod, query-frontend pod fails to gracefully stop in 30s default window. This means that k8s has to kill it viaSIGKILLsignal and then this pod will be hanging inerrorstate. As seen in the logs, multiple modules are stopped but it keep logging message below until it is killedFull log
2025-10-11 02:30:22.086 level=warn ts=2025-10-11T00:30:22.086496213Z caller=server.go:1539 method=/frontend.Frontend/Process duration=332.034µs msg=gRPC err="queue is stopped" 2025-10-11 02:30:22.025 level=warn ts=2025-10-11T00:30:22.025431275Z caller=server.go:1539 method=/frontend.Frontend/Process duration=343.21µs msg=gRPC err="queue is stopped" 2025-10-11 02:30:21.884..... same log multiple times.....
level=warn ts=2025-10-11T00:29:52.165841052Z caller=server.go:1539 method=/frontend.Frontend/Process duration=282.481µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.153
level=warn ts=2025-10-11T00:29:52.153100985Z caller=server.go:1539 method=/frontend.Frontend/Process duration=261.76µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.149
level=warn ts=2025-10-11T00:29:52.149313234Z caller=server.go:1539 method=/frontend.Frontend/Process duration=260.199µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.149
level=warn ts=2025-10-11T00:29:52.149057059Z caller=server.go:1539 method=/frontend.Frontend/Process duration=355.81µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.146
level=warn ts=2025-10-11T00:29:52.14696915Z caller=server.go:1539 method=/frontend.Frontend/Process duration=414.484µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.146
level=warn ts=2025-10-11T00:29:52.146919925Z caller=server.go:1539 method=/frontend.Frontend/Process duration=374.843µs msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093738737Z caller=module_service.go:120 msg="module stopped" module=overrides
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093691152Z caller=module_service.go:120 msg="module stopped" module=memberlist-kv
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093679631Z caller=module_service.go:120 msg="module stopped" module=overrides-api
2025-10-11 02:29:52.093
level=info ts=2025-10-11T00:29:52.093659119Z caller=module_service.go:120 msg="module stopped" module=query-frontend
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093637758Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.031021272s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093621838Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.031005288s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093590661Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040844602s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093473889Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040857282s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093457553Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040658244s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093443544Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040680388s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093427272Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h32m34.040643844s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093412296Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.030680343s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093391703Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.030704815s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.093
level=warn ts=2025-10-11T00:29:52.093245139Z caller=server.go:1539 method=/frontend.Frontend/Process duration=10h31m54.030534339s msg=gRPC err="queue is stopped"
2025-10-11 02:29:52.092
level=info ts=2025-10-11T00:29:52.09272838Z caller=app.go:239 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
To Reproduce
Steps to reproduce the behavior:
Expected behavior
query-fronted should be able to handle termination gracefully and properly exit
Environment:
Additional Context
This issue happened at 2:30 am so there were no queries executed / activity at the time
Perhaps this is also related to #2847