Pin otel-lgtm and rebase the Tempo config onto 3.0

[grey275]

What broke

Tempo is dead in the deployed stack. Grafana can't reach it on :3200 from inside its own container, which rules out the network, the mounts, dockge and the app:

Get "http://127.0.0.1:3200/api/metrics/query_range?..." : dial tcp 127.0.0.1:3200: connect: connection refused

The otel container log is the proof. Every component reports ready except Tempo, which starts and never does:

Starting grafana/otel-lgtm v0.29.0
Running Tempo v3.0.2 logging=false
Prometheus is up and running.   Otelcol is up and running.
Loki is up and running.         Grafana is up and running.
Pyroscope is up and running.

Why

observability/tempo-config.yaml is a verbatim copy of the image's own config, but docker-compose.yaml tracked grafana/otel-lgtm:latest. The image moved to 0.29.0 and took Tempo from 2.x to 3.0.2 underneath the copy. Per the 2.x to 3.0 migration guide, Tempo 3 "refuses to start if it detects legacy (flat, unscoped) overrides" — which is what the copy carried:

overrides:
  metrics_generator_processors: [service-graphs, local-blocks, span-metrics]

Tempo exits on a config error rather than starting with defaults, so nothing else notices: the container is healthy, logs and metrics keep flowing, and only traces go dark.

The vendored copy had also drifted from the image in both directions: it carried an ingester block the image no longer has, and was missing the image's memberlist block. That is exactly the un-configure hazard the file's own header comment warns about.

The fix

• Pin grafana/otel-lgtm:0.29.0. This is the root cause, not the overrides key. The copies under observability/ and the tag are one unit; :latest guarantees a recurrence on some future pull. The README now spells out the re-copy step for the next upgrade.
• Rebase tempo-config.yaml on 0.29.0's, re-applying the 72h retention delta and the local-blocks processor (TraceQL metrics, which Traces Drilldown issues on open) in the scoped format. Top-level keys are now exactly upstream's plus our intended compactor.

loki-config.yaml is untouched: diffed against 0.29.0's and it is still verbatim plus its documented deltas, which is why Loki stayed up.

Still open: the metrics half

The dashboard's slm_* panels are empty and this PR does not explain that. Ruled out so far: the app exports all three signals to one endpoint and logs arrive, so the collector has the metrics; Metrics.setup() is called (src/server/main.ts:113); Prometheus is up; and 0.29.0 sets no otlp.translation_strategy, so the Prometheus 3.x default produces exactly the underscored names the dashboards query. Next check on the host:

docker exec slmgrafana wget -qO- 'http://localhost:9090/api/v1/label/__name__/values' | grep -i slm

Verification

Not verified against a running image (the deploy host is where this reproduces). YAML parses and the diff is grounded in the image's real 0.29.0 config, fetched from the upstream repo. To see Tempo's actual parse error, add ENABLE_LOGS_TEMPO=true to the otel service — logging=false in the log above is otel-lgtm suppressing each component's own output.

🤖 Generated with Claude Code