Fix tempo-config for Tempo 3 (otel-lgtm:0.29.0) - #51
Merged
Conversation
The pinned grafana/otel-lgtm:0.29.0 ships Tempo 3.0.2, which removed the
compactor component (#6273) and the metrics-generator local-blocks
processor (#6555). The committed tempo-config.yaml still carried the
Tempo-2.x forms of both deltas (top-level `compactor`,
`metrics_generator.processor.local_blocks`, `metrics_generator.traces_storage`,
and a `local-blocks` entry in the overrides processors list), so Tempo failed
config parse and exited silently (logging=false): the container stayed
healthy, the other five components reported ready, and the only symptom was
traces never reaching Grafana.
Re-author both deltas for Tempo 3:
- 72h block_retention now lives on `backend_scheduler.provider.compaction.compaction`
and `backend_worker.compaction` instead of a top-level `compactor` block
(verified against Tempo's /status/config; effective value reads 72h).
- Drop the local-blocks processor and traces_storage; TraceQL metrics
(`{...} | rate() by(...)`) are served natively in Tempo 3, confirmed with a
live /api/metrics/query_range call returning series.
Verified end to end: config parses (`tempo -config.verify=true`), Tempo comes
up, traces are searchable, and RED/TraceQL metrics work. Update the
observability README mount-contract row to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Found while running a from-scratch
docs/INSTALLING.mdinstall to vet the Grafana stack: traces never reach Grafana on a clean deploy.The pinned
grafana/otel-lgtm:0.29.0ships Tempo 3.0.2, which removed:But the committed
observability/tempo-config.yamlstill carried the Tempo-2.x forms of both deltas: a top-levelcompactor:block,metrics_generator.processor.local_blocks,metrics_generator.traces_storage, and alocal-blocksentry in the overrides processors list. Tempo fails config parse:and exits silently (
logging=false). The container stays healthy, the other five components report ready, logs and metrics keep flowing — the only symptom is traces going dark. Exactly the failure modeobservability/README.mdwarns about. (PR #37 fixed the overrides format for Tempo 3 but not these three fields.)Fix
Re-author both deltas for Tempo 3:
block_retentionnow lives onbackend_scheduler.provider.compaction.compactionandbackend_worker.compaction(the scheduler/worker that replaced the compactor). The nestedprovider.compaction.compactionpath is confirmed against Tempo's own/status/config— the-helpflag paths are mangled for this struct.traces_storage. TraceQL metrics ({...} | rate() by(...), which the Traces Drilldown issues) are served natively in Tempo 3 without the processor.Also updates the README mount-contract row to match.
Verification
tempo -config.verify=trueon the committed file: parses clean/status/configafter startup:block_retention: 72h0m0son both scheduler and worker/api/metrics/query_rangewith{} | rate()returns seriesslm_*) and logs (Loki) unaffectedNot a frontend change; no dev-server link. Reviewers can see it running in the vetting install at
http://localhost:3001(Grafana admin/admin).🤖 Generated with Claude Code