Breaking Changes

• [CHANGE] BREAKING CHANGE Migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. #5058 (@iamrajiv, @javiermolinar)
• [CHANGE] BREAKING CHANGE TraceQL Metrics buckets are calculated based on data in past. #5366 (@ruslan-mikhailov)
• [CHANGE] BREAKING CHANGE Fix incorrect TraceQL metrics results when series labels include strings and integers with same textural representation.
  This also changes the TraceQL metrics responses of /api/metrics/query_range and /api/metrics/query to remove the redundant
  prom_labels field which was the error source. There may be an interruption to TraceQL metrics queries during rollout while components are running the previous version. #5659 (@mdisibio)
• [CHANGE] BREAKING CHANGE Deprecating vParquet2 block format #5688 (@ie-pham)

Changes

• [CHANGE] Return Bad Request from all frontend endpoints if the tenant can't be extracted. #5480 (@carles-grafana)
• [CHANGE] Do not count cached querier responses for SLO metrics such as inspected bytes. #5185 (@carles-grafana)
• [CHANGE] Adjust the definition of tempo_metrics_generator_processor_service_graphs_expired_edges to exclude edges that are counted in the service graph. #5319 (@joe-elliott)
• [CHANGE] Command tempo-cli analyse block(s) excludes attributes with array values. #5380 (@stoewer)
• [CHANGE] Remove .005s and add a 1.5s bucket to all request duration histograms. #5492 (@joe-elliott)
• [CHANGE] Improve tempo writes dashboard. #5500 (@javiermolinar)
• [CHANGE] Upgrade Tempo to go 1.25.0. #5548 (@javiermolinar)
• [CHANGE] Upgrade Tempo to go 1.25.1 #5685 (@electron0zero)
• [CHANGE] Drop tracing bridges in favor of OTEL only tracing. #5594 (@zalegrala)
• [CHANGE] Enable HTTP writes in the multi-tenant example. #5297 (@carles-grafana)

Features

• [FEATURE] Add MCP Server support. #5212 (@joe-elliott)
• [FEATURE] Add query hints sample=true and sample=0.xx which can speed up TraceQL metrics queries by sampling a subset of the data to provide an approximate result. #5469 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview1 with low-resolution timestamp columns for better TraceQL metrics performance. This format is in development and breaking changes are expected before final release. #5495 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview2 with dedicated attribute columns for integers. This format is in development and breaking changes are expected before final release. #5639 (@stoewer)

Enhancement

• [ENHANCEMENT] Add counter query_frontend_bytes_inspected_total, which shows the total number of bytes read from disk and object storage #5310 (@carles-grafana)
• [ENHANCEMENT] Add histograms spans_distance_in_future_seconds / spans_distance_in_past_seconds that count spans with end timestamp in the future / past. While spans in the future are accepted, they are invalid and may not be found using the Search API. #4936 (@carles-grafana)
• [ENHANCEMENT] Add support for scope in cost-attribution usage tracker. #5646 (@electron0zero)
• [ENHANCEMENT] Add alert for high error rate reported by vulture. #5206 (@ruslan-mikhailov)
• [ENHANCEMENT] Support the new db.namespace attribute for service-graph DB nodes. #5602 (@gouthamve)
• [ENHANCEMENT] TraceQL metrics performance increase for simple queries. #5247 (@mdisibio)
• [ENHANCEMENT] TraceQL search and metrics performance increase. #5280 (@mdisibio)
• [ENHANCEMENT] TraceQL performance improvement. #5218 (@mdisibio)
• [ENHANCEMENT] TraceQL compare() performance improvement. #5419 (@mdisibio)
• [ENHANCEMENT] Align traceql attribute struct for better performance. #5240 (@mdisibio)
• [ENHANCEMENT] Drop invalid prometheus label names in the spanmetrics processor. #5122 (@KyriosGN0)
• [ENHANCEMENT] Improve logging and tracing in the write path to include tenant info. #5436 (@javiermolinar)
• [ENHANCEMENT] Added usage tracker example. #5356 (@javiermolinar)
• [ENHANCEMENT] Add Stop method. #5293 (@stephanos)
• [ENHANCEMENT] Use peer attributes to determine the name of a client service virtual node in the service graph. #5381 (@MartenM)
• [ENHANCEMENT] Put actual size for writing to backend. #5413 (@ruslan-mikhailov)
• [ENHANCEMENT] Upgrade Azurite and Fake-gcs-server to latest version. #5512 (@javiermolinar)
• [ENHANCEMENT] Make block ordering deterministic. #5411 (@rajiv-singh)
• [ENHANCEMENT] Improve exemplar selection in quantile_over_time(). #5278 (@zalegrala)
• [ENHANCEMENT] Measure bytes received before limits and publish it as tempo_distributor_ingress_bytes_total. #5601 (@mapno)
• [ENHANCEMENT] Add total size logging functionality to track trace #5625(@sienna011022)

Bugfix

• [BUGFIX] Fix Tempo configuration options that are always overrided with config overrides section. #5202 (@KyriosGN0)
• [BUGFIX] Correctly apply trace idle period in ingesters and add the concept of trace live period. #5346 (@joe-elliott)
• [BUGFIX] Fix invalid YAML output from /status/runtime_config endpoint by adding document separator. #5371 (@iamrajiv)
• [BUGFIX] Fix panic in query_range HTTP handling that could be triggered by cancellations or other errors. #5667 (@mdisibio)
• [BUGFIX] Do not allow very small steps. #5441 (@ruslan-mikhailov)
• [BUGFIX] Fix incorrect TraceQL string comparison of strings starting with numbers. #5658 (@mdisibio)
• [BUGFIX] Fix incorrect results in TraceQL compare() for spans with array attributes #5519 (@ruslan-mikhailov)
• [BUGFIX] Fix cache collision for incomplete query in SearchTagValuesV2 #5549 (@ruslan-mikhailov)
• [BUGFIX] Fix for structural operator with empty left-hand spanset. #5578 (@ruslan-mikhailov)
• [BUGFIX] Deadlock on invalid query to api/v2/search/tags. (SearchTagsV2) #5607 (@ruslan-mikhailov)
• [BUGFIX] Fixed incorrect root span detection when spans have a child_of link but no parent. #3634 (@mexirica)
• [BUGFIX] Prevent metrics-generator WAL deletion when tenant is empty. #5586 (@sienna011022)
• [BUGFIX] Fix docker-compose port configuration for Alloy gRPC (4319 → 4317). #5536
• [BUGFIX] Fix panic error from empty span id. #5464
• [BUGFIX] Return Bad Request from frontend if the provided tag is invalid in SearchTagValuesV2 endpoint. #5493 (@carles-grafana)

Tempo Rearchitecture [EXPERIMENTAL]

• [CHANGE] BREAKING CHANGE Drop unused backend_scheduler.tenant_measurement_interval, use backend_scheduler.compaction.measure_interval instead. #5328 (@zalegrala)
• [CHANGE] Allow configuration of min/max input blocks for compaction provider. #5373 (@zalegrala)
• [CHANGE] BREAKING CHANGE Add require minimum time between tenant sorting in backend-scheduler. #5410 (@zalegrala)
  The configuration for backend_scheduler.provider.compaction.backoff has been removed.
  Additionally, the compaction_tenant_backoff_total metric has been renamed to compaction_empty_tenant_cycle_total for clarity.
• [CHANGE] Shard backend-scheduler work files locally and modify backend work structure to accommodate sharding approach. #5412 (@zalegrala)
• [CHANGE] Change worker to shutdown after the current job, waiting 30s by default. [#5460](ht...

Breaking Changes

• [CHANGE] BREAKING CHANGE We are no longer publishing rpm and deb packages due to an internal change to the handling of signing keys. This can be restored if we find that folks are actually using these packages. #5684 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. #5058 (@iamrajiv, @javiermolinar)
• [CHANGE] BREAKING CHANGE TraceQL Metrics buckets are calculated based on data in past. #5366 (@ruslan-mikhailov)
• [CHANGE] BREAKING CHANGE Fix incorrect TraceQL metrics results when series labels include strings and integers with same textural representation.
  This also changes the TraceQL metrics responses of /api/metrics/query_range and /api/metrics/query to remove the redundant
  prom_labels field which was the error source. There may be an interruption to TraceQL metrics queries during rollout while components are running the previous version. #5659 (@mdisibio)

Changes

• [CHANGE] Return Bad Request from all frontend endpoints if the tenant can't be extracted. #5480 (@carles-grafana)
• [CHANGE] Do not count cached querier responses for SLO metrics such as inspected bytes. #5185 (@carles-grafana)
• [CHANGE] Adjust the definition of tempo_metrics_generator_processor_service_graphs_expired_edges to exclude edges that are counted in the service graph. #5319 (@joe-elliott)
• [CHANGE] Command tempo-cli analyse block(s) excludes attributes with array values. #5380 (@stoewer)
• [CHANGE] Remove .005s and add a 1.5s bucket to all request duration histograms. #5492 (@joe-elliott)
• [CHANGE] Improve tempo writes dashboard. #5500 (@javiermolinar)
• [CHANGE] Upgrade Tempo to go 1.25.0. #5548 (@javiermolinar)
• [CHANGE] Drop tracing bridges in favor of OTEL only tracing. #5594 (@zalegrala)
• [CHANGE] Enable HTTP writes in the multi-tenant example. #5297 (@carles-grafana)
• [CHANGE] Upgrade Tempo to go 1.25.1 #5685 (@electron0zero)

Features

• [FEATURE] Add MCP Server support. #5212 (@joe-elliott)
• [FEATURE] Add query hints sample=true and sample=0.xx which can speed up TraceQL metrics queries by sampling a subset of the data to provide an approximate result. #5469 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview1 with low-resolution timestamp columns for better TraceQL metrics performance. This format is in development and breaking changes are expected before final release. #5495 (@mdisibio)
• [FEATURE] New block encoding vParquet5-preview2 with dedicated attribute columns for integers. This format is in development and breaking changes are expected before final release. #5639 (@stoewer)

Enhancement

• [ENHANCEMENT] Add counter query_frontend_bytes_inspected_total, which shows the total number of bytes read from disk and object storage #5310 (@carles-grafana)
• [ENHANCEMENT] Add histograms spans_distance_in_future_seconds / spans_distance_in_past_seconds that count spans with end timestamp in the future / past. While spans in the future are accepted, they are invalid and may not be found using the Search API. #4936 (@carles-grafana)
• [ENHANCEMENT] Add support for scope in cost-attribution usage tracker. #5646 (@electron0zero)
• [ENHANCEMENT] Add alert for high error rate reported by vulture. #5206 (@ruslan-mikhailov)
• [ENHANCEMENT] Support the new db.namespace attribute for service-graph DB nodes. #5602 (@gouthamve)
• [ENHANCEMENT] TraceQL metrics performance increase for simple queries. #5247 (@mdisibio)
• [ENHANCEMENT] TraceQL search and metrics performance increase. #5280 (@mdisibio)
• [ENHANCEMENT] TraceQL performance improvement. #5218 (@mdisibio)
• [ENHANCEMENT] TraceQL compare() performance improvement. #5419 (@mdisibio)
• [ENHANCEMENT] Align traceql attribute struct for better performance. #5240 (@mdisibio)
• [ENHANCEMENT] Drop invalid prometheus label names in the spanmetrics processor. #5122 (@KyriosGN0)
• [ENHANCEMENT] Improve logging and tracing in the write path to include tenant info. #5436 (@javiermolinar)
• [ENHANCEMENT] Added usage tracker example. #5356 (@javiermolinar)
• [ENHANCEMENT] Add Stop method. #5293 (@stephanos)
• [ENHANCEMENT] Use peer attributes to determine the name of a client service virtual node in the service graph. #5381 (@MartenM)
• [ENHANCEMENT] Put actual size for writing to backend. #5413 (@ruslan-mikhailov)
• [ENHANCEMENT] Upgrade Azurite and Fake-gcs-server to latest version. #5512 (@javiermolinar)
• [ENHANCEMENT] Make block ordering deterministic. #5411 (@rajiv-singh)
• [ENHANCEMENT] Improve exemplar selection in quantile_over_time(). #5278 (@zalegrala)
• [ENHANCEMENT] Measure bytes received before limits and publish it as tempo_distributor_ingress_bytes_total. #5601 (@mapno)
• [ENHANCEMENT] Add total size logging functionality to track trace #5625(@sienna011022)

Bugfix

• [BUGFIX] Fix Tempo configuration options that are always overrided with config overrides section. #5202 (@KyriosGN0)
• [BUGFIX] Correctly apply trace idle period in ingesters and add the concept of trace live period. #5346 (@joe-elliott)
• [BUGFIX] Fix invalid YAML output from /status/runtime_config endpoint by adding document separator. #5371 (@iamrajiv)
• [BUGFIX] Fix panic in query_range HTTP handling that could be triggered by cancellations or other errors. #5667 (@mdisibio)
• [BUGFIX] Do not allow very small steps. #5441 (@ruslan-mikhailov)
• [BUGFIX] Fix incorrect TraceQL string comparison of strings starting with numbers. #5658 (@mdisibio)
• [BUGFIX] Fix incorrect results in TraceQL compare() for spans with array attributes #5519 (@ruslan-mikhailov)
• [BUGFIX] Fix cache collision for incomplete query in SearchTagValuesV2 #5549 (@ruslan-mikhailov)
• [BUGFIX] Fix for structural operator with empty left-hand spanset. #5578 (@ruslan-mikhailov)
• [BUGFIX] Deadlock on invalid query to api/v2/search/tags. (SearchTagsV2) #5607 (@ruslan-mikhailov)
• [BUGFIX] Fixed incorrect root span detection when spans have a child_of link but no parent. #3634 (@mexirica)
• [BUGFIX] Prevent metrics-generator WAL deletion when tenant is empty. #5586 (@sienna011022)
• [BUGFIX] Fix docker-compose port configuration for Alloy gRPC (4319 → 4317). #5536
• [BUGFIX] Fix panic error from empty span id. #5464
• [BUGFIX] Return Bad Request from frontend if the provided tag is invalid in SearchTagValuesV2 endpoint. #5493 (@carles-grafana)

Tempo Rearchitecture [EXPERIMENTAL]

• [CHANGE] BREAKING CHANGE Drop unused backend_scheduler.tenant_measurement_interval, use backend_scheduler.compaction.measure_interval instead. #5328 (@zalegrala)
• [CHANGE] Allow configuration of min/max input blocks for compaction provider. #5373 (@zalegrala)
• [CHANGE] BREAKING CHANGE Add require minimum time between tenant sorting in backend-scheduler. #5410 (@zalegrala)
  The configuration for backend_scheduler.provider.compaction.backoff has been removed.
  Additionally, the compaction_tenant_backoff_total metric has been renamed to compaction_empty_tenant_cycle_total for clarity.
• [CHANGE] Shard backend-scheduler work files locally and modify backend work structure to accommodate sharding approach. [#5...

• [CHANGE] Update Go to version 1.24.4. #5323 (@stoewer)
• [BUGFIX] Add nil check to partitionAssignmentVar. #5198 (@mapno)
• [BUGFIX] Correct instant query calculation. #5252 (@ruslan-mikhailov)
• [BUGFIX] Fix tracing context propagation in distributor HTTP write requests. #5312 (@mapno)
• [BUGFIX] Fix search by trace:id with short trace ID. #5331 (@ruslan-mikhailov)
• [BUGFIX] Fix bug where most_recent=true wouldn't return most recent results when query overlapped ingesters and few other blocks. #5438 (@joe-elliott)
• [BUGFIX] Fix panic when counter series is missing during avg_over_time aggregation. #5300 (@ie-pham)

• [BUGFIX] Fix ingester issue where a hash collision could lead to spans stored incorrectly #5276 (@carles-grafana)

Breaking Changes

• [CHANGE] BREAKING CHANGE Change default http-listen-port from 80 to 3200 #4960 (@martialblog)
• [CHANGE] BREAKING CHANGE Upgrade OTEL Collector to v0.122.1 #4893 (@javiermolinar)
  The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver
• [CHANGE] BREAKING CHANGE Convert SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it's more performant. #4748 (@carles-grafana)
• [CHANGE] BREAKING CHANGE Remove tempo serverless #4599 (@electron0zero)
  Following config options are no longer valid, please remove them if you are using these in your tempo config:

  querier:
      search:
          prefer_self: <int>
          external_hedge_requests_at: <duration>
          external_hedge_requests_up_to:  <duration>
          external_backend: <string>
          google_cloud_run: <string>
          external_endpoints: <array>
  

  Tempo serverless related metric tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total and tempo_feature_enabled are being removed.
• [CHANGE] BREAKING CHANGE Removed internal_error as a reason from tempo_discarded_spans_total. #4554 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Enforce max attribute size at event, link, and instrumentation scope. Make config per-tenant.
  Renamed max_span_attr_byte to max_attribute_bytes #4633 (@ie-pham)
• [CHANGE] BREAKING CHANGE Removed otel jaeger exporter. #4926 (@javiermolinar)

Changes

• [CHANGE] Update query range error message #4929 (@joey-grafana)
• [CHANGE] Improve rate limit error message when traces size exceeds rate limit #4986 (@electron0zero)
• [CHANGE] Tempo Query: replace opentracing-contrib/go-grpc by otelgrpc. #4958 (@javiermolinar)
• [CHANGE] Improve parquet readers io.ReaderAt compatibility #4963 (@joe-elliott)
• [CHANGE] Finish polling current tenants on poller shutdown #4897 (@zalegrala)
• [CHANGE] Set querier default level to INFO #4943 (@javiermolinar)
• [CHANGE] Change retention to honor compactor disablement #5044 (@zalegrala)
• [CHANGE] Continue on error in tempo-cli rewrite-blocks #5043 (@zalegrala)
• [CHANGE] Update jsonnet memcached and exporter image versions #5056 (@zalegrala)
• [CHANGE] Enforce max series in response for metrics queries with new query-frontend configuration #4525 (@ie-pham)
• [CHANGE] Update to go 1.24.3 #5110 (@stoewer)
• [CHANGE] Update to go 1.24.2 #5057 #5082 (@carsontham)
• [CHANGE] Update to go 1.24.1 #4704 (@ruslan-mikhailov) #4793 (@javiermolinar)

Features

• [FEATURE] Added most_recent=true query hint to TraceQL to return most recent results. #4238 (@joe-elliott)
• [FEATURE] TraceQL metrics: sum_over_time #4786 (@javiermolinar)
• [FEATURE] Add support for topk and bottomk functions for TraceQL metrics #4646 (@electron0zero)
• [FEATURE] TraceQL: add support for querying by parent span id #4692 (@ie-pham)

Enhancements

• [ENHANCEMENT] Add throughput SLO and metrics for the TraceByID endpoint. #4668 (@carles-grafana)
  configurable via the throughput_bytes_slo field, and it will populate op="traces" label in slo and throughput metrics.
• [ENHANCEMENT] Add ability to add artificial delay to push requests #4716 #4899 #5035 (@yvrhdn, @mapno)
• [ENHANCEMENT] tempo-vulture now generates spans with a parent, instead of only root spans #5154 (@carles-grafana)
• [ENHANCEMENT] Add default mutex and blocking values. #4979 (@mattdurham)
• [ENHANCEMENT] Improve Tempo build options #4755 (@stoewer)
• [ENHANCEMENT] Rewrite traces using rebatching #4690 (@stoewer @joe-elliott)
• [ENHANCEMENT] Reorder span iterators #4754 (@stoewer)
• [ENHANCEMENT] Update minio to version #4341 (@javiermolinar)
• [ENHANCEMENT] Prevent queries in the ingester from blocking flushing traces to disk and memory spikes. #4483 (@joe-elliott)
• [ENHANCEMENT] Update tempo operational dashboard for new block-builder and v2 traces api #4559 (@mdisibio)
• [ENHANCEMENT] Improve metrics-generator performance and stability by applying queue back pressure and concurrency #4721 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by flushing blocks concurrently #4565 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance #4596 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by not using WAL stage #4647 #4671 (@mdisibio)
• [ENHANCEMENT] Export new tempo_ingest_group_partition_lag metric from block-builders and metrics-generators #4571 (@mdisibio)
• [ENHANCEMENT] Overall iterator performance improvement by using max definition level to ignore parts of the RowNumber while nexting. #4753 (@joe-elliott)
• [ENHANCEMENT] Use distroless base container images for improved security #4556 (@carles-grafana)
• [ENHANCEMENT] Rhythm: add block builder to resources dashboard #4556 (@javiermolinar)
• [ENHANCEMENT] Upgrade prometheus to version 3.1.0 #4805 (@javiermolinar)
• [ENHANCEMENT] Update dskit to latest version #4681 (@javiermolinar) #4865 (@javiermolinar)
• [ENHANCEMENT] Increase query-frontend default batch size #4844 (@javiermolinar)
• [ENHANCEMENT] Improve TraceQL perf by reverting EqualRowNumber to an inlineable function.#4705 (@joe-elliott)
• [ENHANCEMENT] Rhythm: Implement MaxBytesPerCycle #4835 (@javiermolinar)
• [ENHANCEMENT] Rhythm: fair partition consumption in blockbuilders #4655 (@javiermolinar)
• [ENHANCEMENT] Rhythm: retry on commit error #4874 (@javiermolinar)
• [ENHANCEMENT] Skip creating one span-traces for every pushed spans in metrics generator #4844 (@javiermolinar)
• [ENHANCEMENT] Improve Tempo / Writes dashboard by adding a kafka panel #4947 (@javiermolinar)
• [ENHANCEMENT] Improve memcached memory usage by pooling buffers #4970 (@joe-elliott)
• [ENHANCEMENT] metrics-generator: allow skipping localblocks and consuming from a different source of data #4686 (@flxbk)
• [ENHANCEMENT] compactor: restore dedicated columns logging for completed blocks #4832 (@edgarkz)
• [ENHANCEMENT] Compactor: pooling changes to reduce memory usage #4985 (@mdisibio)
• [ENHANCEMENT] distributor: add IPv6 support #4840 (@gjacquet)
• [ENHANCEMENT] Support TraceQL Metrics checks in Vulture #4886 (@ruslan-mikhailov)
• [ENHANCEMENT] Add memcached to the resources dashboard #5049 (@javiermolinar)
• [ENHANCEMENT] Include partition owned metric for blockbuilders #5042 (@javiermolinar)
• [ENHANCEMENT] Query-frontend: logs add msg to the log line #4975 (@jmichalek132)
• [ENHANCEMENT] Host Info Processor: track host identifying resource attribute in metric #5152 (@rlankfo)
• [ENHANCEMENT] Vulture checks recent traces #5157 (@ruslan-mikhailov)
• [ENHANCEMENT] Add jitter in backendworker to avoid thundering herd from workers [#515...

Enhancements

• [ENHANCEMENT] TraceQL Metrics: distribute exemplars over time #5158 (@ruslan-mikhailov)
• [ENHANCEMENT] TraceQL Metrics: hard limit number of exemplars #5158 (@ruslan-mikhailov)

Bugfixes

• [BUGFIX] Excluded nestedSetParent and other values from compare() function #5196 (@mdisibio)
• [BUGFIX] Fix distributor issue where a hash collision could lead to spans stored incorrectly #5186 (@mdisibio)
• [BUGFIX] Fix structural metrics rate by aggregation #5204 (@zalegrala)
• [BUGFIX] TraceQL Metrics: right exemplars for histogram and quantiles #5145 (@ruslan-mikhailov)

Breaking Changes

• [CHANGE] BREAKING CHANGE Change default http-listen-port from 80 to 3200 #4960 (@martialblog)
• [CHANGE] BREAKING CHANGE Upgrade OTEL Collector to v0.122.1 #4893 (@javiermolinar)
  The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver
• [CHANGE] BREAKING CHANGE Convert SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it's more performant. #4748 (@carles-grafana)
• [CHANGE] BREAKING CHANGE Remove tempo serverless #4599 (@electron0zero)
  Following config options are no longer valid, please remove them if you are using these in your tempo config:

  querier:
      search:
          prefer_self: <int>
          external_hedge_requests_at: <duration>
          external_hedge_requests_up_to:  <duration>
          external_backend: <string>
          google_cloud_run: <string>
          external_endpoints: <array>
  

  Tempo serverless related metric tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total and tempo_feature_enabled are being removed.
• [CHANGE] BREAKING CHANGE Removed internal_error as a reason from tempo_discarded_spans_total. #4554 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Enforce max attribute size at event, link, and instrumentation scope. Make config per-tenant.
  Renamed max_span_attr_byte to max_attribute_bytes #4633 (@ie-pham)
• [CHANGE] BREAKING CHANGE Removed otel jaeger exporter. #4926 (@javiermolinar)

Changes

• [CHANGE] Update query range error message #4929 (@joey-grafana)
• [CHANGE] Improve rate limit error message when traces size exceeds rate limit #4986 (@electron0zero)
• [CHANGE] Tempo Query: replace opentracing-contrib/go-grpc by otelgrpc. #4958 (@javiermolinar)
• [CHANGE] Improve parquet readers io.ReaderAt compatibility #4963 (@joe-elliott)
• [CHANGE] Finish polling current tenants on poller shutdown #4897 (@zalegrala)
• [CHANGE] Set querier default level to INFO #4943 (@javiermolinar)
• [CHANGE] Change retention to honor compactor disablement #5044 (@zalegrala)
• [CHANGE] Continue on error in tempo-cli rewrite-blocks #5043 (@zalegrala)
• [CHANGE] Update jsonnet memcached and exporter image versions #5056 (@zalegrala)
• [CHANGE] Enforce max series in response for metrics queries with new query-frontend configuration #4525 (@ie-pham)
• [CHANGE] Update to go 1.24.3 #5110 (@stoewer)
• [CHANGE] Update to go 1.24.2 #5057 #5082 (@carsontham)
• [CHANGE] Update to go 1.24.1 #4704 (@ruslan-mikhailov) #4793 (@javiermolinar)

Features

• [FEATURE] Added most_recent=true query hint to TraceQL to return most recent results. #4238 (@joe-elliott)
• [FEATURE] TraceQL metrics: sum_over_time #4786 (@javiermolinar)
• [FEATURE] Add support for topk and bottomk functions for TraceQL metrics #4646 (@electron0zero)
• [FEATURE] TraceQL: add support for querying by parent span id #4692 (@ie-pham)

Enhancements

• [ENHANCEMENT] Add throughput SLO and metrics for the TraceByID endpoint. #4668 (@carles-grafana)
  configurable via the throughput_bytes_slo field, and it will populate op="traces" label in slo and throughput metrics.
• [ENHANCEMENT] Add ability to add artificial delay to push requests #4716 #4899 #5035 (@yvrhdn, @mapno)
• [ENHANCEMENT] tempo-vulture now generates spans with a parent, instead of only root spans #5154 (@carles-grafana)
• [ENHANCEMENT] Add default mutex and blocking values. #4979 (@mattdurham)
• [ENHANCEMENT] Improve Tempo build options #4755 (@stoewer)
• [ENHANCEMENT] Rewrite traces using rebatching #4690 (@stoewer @joe-elliott)
• [ENHANCEMENT] Reorder span iterators #4754 (@stoewer)
• [ENHANCEMENT] Update minio to version #4341 (@javiermolinar)
• [ENHANCEMENT] Prevent queries in the ingester from blocking flushing traces to disk and memory spikes. #4483 (@joe-elliott)
• [ENHANCEMENT] Update tempo operational dashboard for new block-builder and v2 traces api #4559 (@mdisibio)
• [ENHANCEMENT] Improve metrics-generator performance and stability by applying queue back pressure and concurrency #4721 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by flushing blocks concurrently #4565 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance #4596 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by not using WAL stage #4647 #4671 (@mdisibio)
• [ENHANCEMENT] Export new tempo_ingest_group_partition_lag metric from block-builders and metrics-generators #4571 (@mdisibio)
• [ENHANCEMENT] Overall iterator performance improvement by using max definition level to ignore parts of the RowNumber while nexting. #4753 (@joe-elliott)
• [ENHANCEMENT] Use distroless base container images for improved security #4556 (@carles-grafana)
• [ENHANCEMENT] Rhythm: add block builder to resources dashboard #4556 (@javiermolinar)
• [ENHANCEMENT] Upgrade prometheus to version 3.1.0 #4805 (@javiermolinar)
• [ENHANCEMENT] Update dskit to latest version #4681 (@javiermolinar) #4865 (@javiermolinar)
• [ENHANCEMENT] Increase query-frontend default batch size #4844 (@javiermolinar)
• [ENHANCEMENT] Improve TraceQL perf by reverting EqualRowNumber to an inlineable function.#4705 (@joe-elliott)
• [ENHANCEMENT] Rhythm: Implement MaxBytesPerCycle #4835 (@javiermolinar)
• [ENHANCEMENT] Rhythm: fair partition consumption in blockbuilders #4655 (@javiermolinar)
• [ENHANCEMENT] Rhythm: retry on commit error #4874 (@javiermolinar)
• [ENHANCEMENT] Skip creating one span-traces for every pushed spans in metrics generator #4844 (@javiermolinar)
• [ENHANCEMENT] Improve Tempo / Writes dashboard by adding a kafka panel #4947 (@javiermolinar)
• [ENHANCEMENT] Improve memcached memory usage by pooling buffers #4970 (@joe-elliott)
• [ENHANCEMENT] metrics-generator: allow skipping localblocks and consuming from a different source of data #4686 (@flxbk)
• [ENHANCEMENT] compactor: restore dedicated columns logging for completed blocks #4832 (@edgarkz)
• [ENHANCEMENT] Compactor: pooling changes to reduce memory usage #4985 (@mdisibio)
• [ENHANCEMENT] distributor: add IPv6 support #4840 (@gjacquet)
• [ENHANCEMENT] Support TraceQL Metrics checks in Vulture #4886 (@ruslan-mikhailov)
• [ENHANCEMENT] Add memcached to the resources dashboard #5049 (@javiermolinar)
• [ENHANCEMENT] Include partition owned metric for blockbuilders #5042 (@javiermolinar)
• [ENHANCEMENT] Query-frontend: logs add msg to the log line #4975 (@jmichalek132)
• [ENHANCEMENT] Host Info Processor: track host identifying resource attribute in metric #5152 (@rlankfo)
• [ENHANCEMENT] Vulture checks recent traces #5157 (@ruslan-mikhailov)
• [ENHANCEMENT] Add jitter in backendworker to avoid thundering herd from workers [#515...

Bugfixes

• [BUGFIX] Fix rare panic that occurred when a querier modified results from ingesters/generators while they were being marshalled to proto. #4790 (@joe-elliott)
  This bug also impacted query correctness on recent trace data by returning incomplete results before they were ready.

Thanks to @pavleec for filing an issue bringing the query correctness element of this bug to our attention as well as testing a fix.

Changes

• [CHANGE] Default to snappy compression for all gRPC communications internal to Tempo. We feel this is a nice balance of resource usage and network traffic. For a discussion on alternatives see https://github.com/grafana/tempo/discussions/4683. #4696 (@joe-elliott)

Thanks to @edgarkz for championing this issue and providing details of the various compression settings on their install.

Deprecations

• Tempo serverless features are now deprecated and will be removed in an upcoming release #4017 @electron0zero

Breaking Changes

• Added maximum spans per span set to prevent queries from overwhelming read path. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger
• Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
• Removed querier_forget_delay setting from the frontend. This configuration option did nothing. #3996 (@joe-elliott)
• Use Prometheus fast regexp for TraceQL regular expression matchers. #4329 (@joe-elliott)
  All regular expression matchers will now be fully anchored. span.foo =~ "bar" will now be evaluated as span.foo =~ "^bar$"

Changes

• [CHANGE] Disable gRPC compression in the querier and distributor for performance reasons #4429 (@carles-grafana)
  If you would like to re-enable it, we recommend 'snappy'. Use the following settings:

ingester_client:
    grpc_client_config:
        grpc_compression: "snappy"
metrics_generator_client:
    grpc_client_config:
        grpc_compression: "snappy"
querier:
    frontend_worker:
        grpc_client_config:
            grpc_compression: "snappy"

• [CHANGE] slo: include request cancellations within SLO [#4355] (#4355) (@electron0zero)
  request cancellations are exposed under result label in tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total with completed or canceled values to differentiate between completed and canceled requests.
• [CHANGE] update default config values to better align with production workloads #4340 (@electron0zero)
• [CHANGE] tempo-cli: add support for /api/v2/traces endpoint #4127 (@electron0zero)
  BREAKING CHANGE The tempo-cli now uses the /api/v2/traces endpoint by default,
  please use --v1 flag to use /api/traces endpoint, which was the default in previous versions.
• [CHANGE] TraceByID: don't allow concurrent_shards greater than query_shards. #4074 (@electron0zero)
• [CHANGE] BREAKING CHANGE The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• [CHANGE] BREAKING CHANGE Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger.
• [CHANGE] No longer send the final diff in GRPC streaming. Instead we rely on the streamed intermediate results. #4062 (@joe-elliott)
• [CHANGE] Update Go to 1.23.3 #4146 #4147 #4380 (@javiermolinar @mdisibio)
• [CHANGE] Return 422 for TRACE_TOO_LARGE queries #4160 (@zalegrala)
• [CHANGE] Tighten file permissions #4251 (@zalegrala)
• [CHANGE] Drop max live traces log message and rate limit trace too large. #4418 (@joe-elliott)
• [CHANGE] Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  BREAKING CHANGE After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

Features

• [FEATURE] tempo-cli: support dropping multiple traces in a single operation #4266 (@ndk)
• [FEATURE] Discarded span logging log_discarded_spans #3957 (@dastrobu)
• [FEATURE] TraceQL support for instrumentation scope #3967 (@ie-pham)
• [FEATURE] Export cost attribution usage metrics from distributor #4162 (@mdisibio)
• [FEATURE] TraceQL metrics: avg_over_time #4073 (@javiermolinar)
• [FEATURE] TraceQL metrics queries: add min_over_time #3975 (@javiermolinar)
• [FEATURE] TraceQL metrics queries: add max_over_time #4065 (@javiermolinar)
• [FEATURE] Limit tags and tag values search #4320 (@javiermolinar)

Enhancements

• [ENHANCEMENT] TraceQL: Add range condition for byte predicates #4198 (@ie-pham)
• [ENHANCEMENT] Add throughput and SLO metrics in the tags and tag values endpoints #4148 (@electron0zero)
• [ENHANCEMENT] BREAKING CHANGE Add maximum spans per span set. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• [ENHANCEMENT] Add query-frontend limit for max length of query expression #4397 (@electron0zero)
• [ENHANCEMENT] distributor: return trace id length when it is invalid #4407 (@carles-grafana)
• [ENHANCEMENT] Changed log level from INFO to DEBUG for the TempoDB Find operation using traceId to reduce excessive/unwanted logs in log search. #4179 (@Aki0x137)
• [ENHANCEMENT] Pushdown collection of results from generators in the querier #4119 (@electron0zero)
• [ENHANCEMENT] The span multiplier now also sources its value from the resource attributes. #4210 (@JunhoeKim)
• [ENHANCEMENT] TraceQL: Attribute iterators collect matched array values #3867 (@electron0zero, @stoewer)
• [ENHANCEMENT] Allow returning partial traces that exceed the MaxBytes limit for V2 #3941 (@javiermolinar)
• [ENHANCEMENT] Added new middleware to validate request query values #3993 (@javiermolinar)
• [ENHANCEMENT] Prevent massive allocations in the frontend if there is not sufficient pressure from the query pipeline. #3996 (@joe-elliott)
  BREAKING CHANGE Removed querier_forget_delay setting from the frontend. This configuration option did nothing.
• [ENHANCEMENT] Update metrics-generator config in Tempo distributed docker compose example to serve TraceQL metrics #4003 (@javiermolinar)
• [ENHANCEMENT] Reduce allocs related to marshalling dedicated columns repeatedly in the query frontend. #4007 (@joe-elliott)
• [ENHANCEMENT] Improve performance of TraceQL queries #4114 (@mdisibio)
• [ENHANCEMENT] Improve performance of TraceQL queries #4163 (@mdisibio)
• [ENHANCEMENT] Improve performance of some TraceQL queries using select() operation #4438 (@mdisibio)
• [ENHANCEMENT] Reduce memory usage of classic histograms in the span-metrics and service-graphs processors #4232 (@mdisibio)
• [ENHANCEMENT] ...