Skip to content

Fix tempo-config for Tempo 3 (otel-lgtm:0.29.0) - #51

Merged
grey275 merged 1 commit into
mainfrom
worktree-tempo3-config
Jul 17, 2026
Merged

Fix tempo-config for Tempo 3 (otel-lgtm:0.29.0)#51
grey275 merged 1 commit into
mainfrom
worktree-tempo3-config

Conversation

@grey275

@grey275 grey275 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Problem

Found while running a from-scratch docs/INSTALLING.md install to vet the Grafana stack: traces never reach Grafana on a clean deploy.

The pinned grafana/otel-lgtm:0.29.0 ships Tempo 3.0.2, which removed:

  • the compactor component (#6273)
  • the metrics-generator local-blocks processor (#6555)

But the committed observability/tempo-config.yaml still carried the Tempo-2.x forms of both deltas: a top-level compactor: block, metrics_generator.processor.local_blocks, metrics_generator.traces_storage, and a local-blocks entry in the overrides processors list. Tempo fails config parse:

line 39: field compactor not found in type app.Config
line 46: field local_blocks not found in type generator.ProcessorConfig
line 53: field traces_storage not found in type generator.Config

and exits silently (logging=false). The container stays healthy, the other five components report ready, logs and metrics keep flowing — the only symptom is traces going dark. Exactly the failure mode observability/README.md warns about. (PR #37 fixed the overrides format for Tempo 3 but not these three fields.)

Fix

Re-author both deltas for Tempo 3:

  • 72h block_retention now lives on backend_scheduler.provider.compaction.compaction and backend_worker.compaction (the scheduler/worker that replaced the compactor). The nested provider.compaction.compaction path is confirmed against Tempo's own /status/config — the -help flag paths are mangled for this struct.
  • Drop the local-blocks processor and traces_storage. TraceQL metrics ({...} | rate() by(...), which the Traces Drilldown issues) are served natively in Tempo 3 without the processor.

Also updates the README mount-contract row to match.

Verification

  • tempo -config.verify=true on the committed file: parses clean
  • /status/config after startup: block_retention: 72h0m0s on both scheduler and worker
  • Applied to a live install: Tempo comes up ready, traces are searchable, /api/metrics/query_range with {} | rate() returns series
  • Metrics (Prometheus slm_*) and logs (Loki) unaffected

Not a frontend change; no dev-server link. Reviewers can see it running in the vetting install at http://localhost:3001 (Grafana admin/admin).

🤖 Generated with Claude Code

The pinned grafana/otel-lgtm:0.29.0 ships Tempo 3.0.2, which removed the
compactor component (#6273) and the metrics-generator local-blocks
processor (#6555). The committed tempo-config.yaml still carried the
Tempo-2.x forms of both deltas (top-level `compactor`,
`metrics_generator.processor.local_blocks`, `metrics_generator.traces_storage`,
and a `local-blocks` entry in the overrides processors list), so Tempo failed
config parse and exited silently (logging=false): the container stayed
healthy, the other five components reported ready, and the only symptom was
traces never reaching Grafana.

Re-author both deltas for Tempo 3:
- 72h block_retention now lives on `backend_scheduler.provider.compaction.compaction`
  and `backend_worker.compaction` instead of a top-level `compactor` block
  (verified against Tempo's /status/config; effective value reads 72h).
- Drop the local-blocks processor and traces_storage; TraceQL metrics
  (`{...} | rate() by(...)`) are served natively in Tempo 3, confirmed with a
  live /api/metrics/query_range call returning series.

Verified end to end: config parses (`tempo -config.verify=true`), Tempo comes
up, traces are searchable, and RED/TraceQL metrics work. Update the
observability README mount-contract row to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@grey275
grey275 marked this pull request as ready for review July 17, 2026 17:08
@grey275
grey275 merged commit d935631 into main Jul 17, 2026
1 check passed
@grey275
grey275 deleted the worktree-tempo3-config branch July 27, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant