chore(deps): tempo 1.23.3 → 1.24.4

[Gedulis12]

Bump

Component	Current	→	Target	Risk
tempo (chart)	1.23.3	→	1.24.4	🟡
tempo (app)	2.8.2	→	2.9.2	🟡

Changes

kustomization.yaml — chart version only, no values.yaml changes.

Breaking Changes / Upgrade Notes

This chart bump upgrades Tempo from 2.8.2 → 2.9.0 (with patch releases 2.9.1 and 2.9.2 included in chart 1.24.4).

Tempo 2.9 upgrade considerations

• Bucket calculation changes: TraceQL metrics buckets are now calculated based on data in the past instead of the future, aligning with Prometheus behavior. Fixes empty last-bucket issues. May cause differences in existing dashboards and alerts that rely on TraceQL metrics bucket calculations. (PR 5366)
• Series label handling improvements: The prom_labels field has been removed from TraceQL metrics responses to fix incorrect results when series labels include both strings and integers with the same textual representation (e.g. "500" vs 500). Brief interruptions to TraceQL metrics queries possible during rollout while components run different versions. (PR 5659)
• vParquet5 block format (experimental): New experimental block format with two previews (vParquet5-preview1 — low-resolution timestamp columns; vParquet5-preview2 — dedicated integer columns). Breaking changes expected before final release. Not enabled by default. (PR 5495, PR 5639)
• Go version upgrade: Tempo 2.9.0 upgraded to Go 1.25.0; subsequent patch releases (2.9.2) further upgraded to Go 1.26.2 to address CVEs. (PR 5548)
• RPM/DEB packages discontinued: Tempo no longer publishes RPM and DEB packages (internal signing key change). Not relevant for our Helm-based deployment. (PR 5684)
• Vulture/integration tests migrated to OTLP exporter: Migrated from deprecated Jaeger agent/exporter to standard OTLP gRPC endpoint. Not relevant for our deployment. (PR 5058)
• New default max_result_limit: Search endpoint now defaults to max_result_limit: 262144 (256×1024) instead of 0 (unbounded), addressing CVE-2026-21728. (PR 6525)

Chart-level considerations (1.23.3 → 1.24.4)

No chart-level breaking changes apply — we are already past the 1.21.1 (port 3100→3200) and 1.17.0 (Tempo 2.7) breaking changes. This is a minor bump within the 1.24.x series.

Note: Chart version 1.24.4 (Jan 30, 2026) was the final release in the grafana/helm-charts repository. All subsequent Tempo chart releases are published under grafana-community/helm-charts. Future bumps will need to update the chart repo URL.

Cross-reference with our values.yaml

Our current values.yaml configures:

• extraVolumes / extraVolumeMounts for PVC-backed storage at /var/tempo
• metricsGenerator.enabled: true with remote write to Prometheus
• retention: "72h"
• overrides.defaults.metrics_generator.processors: [local-blocks]

None of these settings interact with the breaking changes above. The MCP server (new in 2.9) is disabled by default and does not require configuration changes. The metrics-generator continues to work with existing local-blocks processor config.

Impact on Current Setup

Not affected

• Bucket calculation changes: Using default retention with no custom buckets — unaffected.
• prom_labels field removal: Not referenced in our configuration — no impact.
• vParquet5 experimental block format: Not enabled (staying on default) — no action needed.
• Go version upgrade / RPM/DEB packaging changes: Transparent in Helm-based deployment.

Beneficial

• Security fixes: Multiple CVEs addressed (Go, gRPC, OTel, x/crypto, expr-lang, xpath, jsonparser, go-jose, etc.) — direct security improvement.
• New default max_result_limit: Addresses CVE-2026-21728 — beneficial by default.

Changelog

Tempo 2.9 Highlights

• MCP (Model Context Protocol) server (experimental): LLM integration for querying tracing data via TraceQL — disabled by default, opt-in per tenant. Provides AI assistants (Claude Code, Cursor) with direct access to tracing data. (PR 5212)
• TraceQL metrics sampling: Adaptive probabilistic sampling via with(sample=true) for improved performance; configurable with with(span_sample=0.xx) or with(trace_sample=0.xx). (PR 5469)
• SLO metrics improvements: Cached querier responses excluded from SLO metrics (inspected bytes), providing more accurate I/O measurements. (PR 5185)
• New metrics: query_frontend_bytes_inspected_total (total bytes read from disk/object store), spans_distance_in_future_seconds / spans_distance_in_past_seconds (span timestamp range histograms), tempo_distributor_ingress_bytes_total (pre-limits ingress volume).
• Metrics-generator improvements: Invalid Prometheus label names auto-dropped, db.namespace attribute support, improved service graph virtual nodes via peer attributes, adjusted expired-edges counter.
• TraceQL performance improvements: Batch processing optimizations, complex query improvements, compare() function optimization, struct alignment for memory performance.
• Cost-attribution tracker: Added scope support (resource. / span. prefixed attributes) for more precise tracking.
• Project Rhythm: New architecture foundation (Kafka-based, live-store, block-builder) addressing HA, write reliability, and TCO. Not yet GA.

Bug fixes (selected)

• Fixed quantile_over_time() exemplar selection
• Fixed TraceQL string comparison for strings starting with numbers
• Fixed incorrect compare() results for spans with array attributes
• Fixed structural operator behavior with empty left-hand spansets
• Fixed query_range panic on cancellation
• Fixed cache collision in SearchTagValuesV2
• Fixed deadlock on invalid SearchTagsV2 query
• Fixed incorrect root span detection (child_of link but no parent)
• Fixed metrics-generator WAL deletion when tenant is empty
• Fixed empty span ID panic
• Fixed configuration options always overridden by overrides section

Security fixes

2.9.1:

• Go updated to 1.25.5 (CVE-2025-61729, CVE-2025-47907, CVE-2025-58183, CVE-2025-61727)
• golang.org/x/crypto updated (CVE-2025-47914, CVE-2025-58181)
• github.com/expr-lang/expr updated to v1.17.7 (CVE-2025-68156)

2.9.2:

• Go updated to 1.26.2 (CVE-2026-25679)
• go.opentelemetry.io/otel/sdk updated to v1.43.0 (CVE-2026-39883)
• OTLP HTTP exporter modules updated (CVE-2026-39882)
• google.golang.org/grpc updated to v1.79.3 (CVE-2026-33186)
• github.com/go-jose/go-jose/v4 updated (CVE-2026-34986)
• github.com/antchfx/xpath updated (CVE-2026-32287)
• github.com/buger/jsonparser updated (GHSA-6g7g-w4f8-9c9x)
• Exemplar hint capped to prevent unbounded memory allocation (CVE-2026-27878)
• Default max_result_limit set to 262144 (CVE-2026-21728)

Chart versions between 1.23.3 and 1.24.4

Chart	Date	App Version	Notes
1.24.0	Oct 26, 2025	2.9.0	Initial Tempo 2.9 release
1.24.1	Nov 21, 2025	2.9.1	Security fixes (Go, x/crypto, expr-lang)
1.24.2	Jan 12, 2026	2.9.2	Security fixes (Go, gRPC, OTel, multiple CVEs)
1.24.3	Jan 13, 2026	2.9.2	Patch (no app version change)
1.24.4	Jan 30, 2026	2.9.2	Final release in grafana/helm-charts; chart repo migrated after this

Source

• https://grafana.com/docs/tempo/latest/release-notes/version-2/v2-9/
• https://github.com/grafana/helm-charts/releases
• https://github.com/grafana/tempo/releases