• [BUGFIX] Fix ingester issue where a hash collision could lead to spans stored incorrectly #5276 (@carles-grafana)

Breaking Changes

• [CHANGE] BREAKING CHANGE Change default http-listen-port from 80 to 3200 #4960 (@martialblog)
• [CHANGE] BREAKING CHANGE Upgrade OTEL Collector to v0.122.1 #4893 (@javiermolinar)
  The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver
• [CHANGE] BREAKING CHANGE Convert SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it's more performant. #4748 (@carles-grafana)
• [CHANGE] BREAKING CHANGE Remove tempo serverless #4599 (@electron0zero)
  Following config options are no longer valid, please remove them if you are using these in your tempo config:

  querier:
      search:
          prefer_self: <int>
          external_hedge_requests_at: <duration>
          external_hedge_requests_up_to:  <duration>
          external_backend: <string>
          google_cloud_run: <string>
          external_endpoints: <array>
  

  Tempo serverless related metric tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total and tempo_feature_enabled are being removed.
• [CHANGE] BREAKING CHANGE Removed internal_error as a reason from tempo_discarded_spans_total. #4554 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Enforce max attribute size at event, link, and instrumentation scope. Make config per-tenant.
  Renamed max_span_attr_byte to max_attribute_bytes #4633 (@ie-pham)
• [CHANGE] BREAKING CHANGE Removed otel jaeger exporter. #4926 (@javiermolinar)

Changes

• [CHANGE] Update query range error message #4929 (@joey-grafana)
• [CHANGE] Improve rate limit error message when traces size exceeds rate limit #4986 (@electron0zero)
• [CHANGE] Tempo Query: replace opentracing-contrib/go-grpc by otelgrpc. #4958 (@javiermolinar)
• [CHANGE] Improve parquet readers io.ReaderAt compatibility #4963 (@joe-elliott)
• [CHANGE] Finish polling current tenants on poller shutdown #4897 (@zalegrala)
• [CHANGE] Set querier default level to INFO #4943 (@javiermolinar)
• [CHANGE] Change retention to honor compactor disablement #5044 (@zalegrala)
• [CHANGE] Continue on error in tempo-cli rewrite-blocks #5043 (@zalegrala)
• [CHANGE] Update jsonnet memcached and exporter image versions #5056 (@zalegrala)
• [CHANGE] Enforce max series in response for metrics queries with new query-frontend configuration #4525 (@ie-pham)
• [CHANGE] Update to go 1.24.3 #5110 (@stoewer)
• [CHANGE] Update to go 1.24.2 #5057 #5082 (@carsontham)
• [CHANGE] Update to go 1.24.1 #4704 (@ruslan-mikhailov) #4793 (@javiermolinar)

Features

• [FEATURE] Added most_recent=true query hint to TraceQL to return most recent results. #4238 (@joe-elliott)
• [FEATURE] TraceQL metrics: sum_over_time #4786 (@javiermolinar)
• [FEATURE] Add support for topk and bottomk functions for TraceQL metrics #4646 (@electron0zero)
• [FEATURE] TraceQL: add support for querying by parent span id #4692 (@ie-pham)

Enhancements

• [ENHANCEMENT] Add throughput SLO and metrics for the TraceByID endpoint. #4668 (@carles-grafana)
  configurable via the throughput_bytes_slo field, and it will populate op="traces" label in slo and throughput metrics.
• [ENHANCEMENT] Add ability to add artificial delay to push requests #4716 #4899 #5035 (@yvrhdn, @mapno)
• [ENHANCEMENT] tempo-vulture now generates spans with a parent, instead of only root spans #5154 (@carles-grafana)
• [ENHANCEMENT] Add default mutex and blocking values. #4979 (@mattdurham)
• [ENHANCEMENT] Improve Tempo build options #4755 (@stoewer)
• [ENHANCEMENT] Rewrite traces using rebatching #4690 (@stoewer @joe-elliott)
• [ENHANCEMENT] Reorder span iterators #4754 (@stoewer)
• [ENHANCEMENT] Update minio to version #4341 (@javiermolinar)
• [ENHANCEMENT] Prevent queries in the ingester from blocking flushing traces to disk and memory spikes. #4483 (@joe-elliott)
• [ENHANCEMENT] Update tempo operational dashboard for new block-builder and v2 traces api #4559 (@mdisibio)
• [ENHANCEMENT] Improve metrics-generator performance and stability by applying queue back pressure and concurrency #4721 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by flushing blocks concurrently #4565 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance #4596 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by not using WAL stage #4647 #4671 (@mdisibio)
• [ENHANCEMENT] Export new tempo_ingest_group_partition_lag metric from block-builders and metrics-generators #4571 (@mdisibio)
• [ENHANCEMENT] Overall iterator performance improvement by using max definition level to ignore parts of the RowNumber while nexting. #4753 (@joe-elliott)
• [ENHANCEMENT] Use distroless base container images for improved security #4556 (@carles-grafana)
• [ENHANCEMENT] Rhythm: add block builder to resources dashboard #4556 (@javiermolinar)
• [ENHANCEMENT] Upgrade prometheus to version 3.1.0 #4805 (@javiermolinar)
• [ENHANCEMENT] Update dskit to latest version #4681 (@javiermolinar) #4865 (@javiermolinar)
• [ENHANCEMENT] Increase query-frontend default batch size #4844 (@javiermolinar)
• [ENHANCEMENT] Improve TraceQL perf by reverting EqualRowNumber to an inlineable function.#4705 (@joe-elliott)
• [ENHANCEMENT] Rhythm: Implement MaxBytesPerCycle #4835 (@javiermolinar)
• [ENHANCEMENT] Rhythm: fair partition consumption in blockbuilders #4655 (@javiermolinar)
• [ENHANCEMENT] Rhythm: retry on commit error #4874 (@javiermolinar)
• [ENHANCEMENT] Skip creating one span-traces for every pushed spans in metrics generator #4844 (@javiermolinar)
• [ENHANCEMENT] Improve Tempo / Writes dashboard by adding a kafka panel #4947 (@javiermolinar)
• [ENHANCEMENT] Improve memcached memory usage by pooling buffers #4970 (@joe-elliott)
• [ENHANCEMENT] metrics-generator: allow skipping localblocks and consuming from a different source of data #4686 (@flxbk)
• [ENHANCEMENT] compactor: restore dedicated columns logging for completed blocks #4832 (@edgarkz)
• [ENHANCEMENT] Compactor: pooling changes to reduce memory usage #4985 (@mdisibio)
• [ENHANCEMENT] distributor: add IPv6 support #4840 (@gjacquet)
• [ENHANCEMENT] Support TraceQL Metrics checks in Vulture #4886 (@ruslan-mikhailov)
• [ENHANCEMENT] Add memcached to the resources dashboard #5049 (@javiermolinar)
• [ENHANCEMENT] Include partition owned metric for blockbuilders #5042 (@javiermolinar)
• [ENHANCEMENT] Query-frontend: logs add msg to the log line #4975 (@jmichalek132)
• [ENHANCEMENT] Host Info Processor: track host identifying resource attribute in metric #5152 (@rlankfo)
• [ENHANCEMENT] Vulture checks recent traces #5157 (@ruslan-mikhailov)
• [ENHANCEMENT] Add jitter in backendworker to avoid thundering herd from workers [#515...

Enhancements

• [ENHANCEMENT] TraceQL Metrics: distribute exemplars over time #5158 (@ruslan-mikhailov)
• [ENHANCEMENT] TraceQL Metrics: hard limit number of exemplars #5158 (@ruslan-mikhailov)

Bugfixes

• [BUGFIX] Excluded nestedSetParent and other values from compare() function #5196 (@mdisibio)
• [BUGFIX] Fix distributor issue where a hash collision could lead to spans stored incorrectly #5186 (@mdisibio)
• [BUGFIX] Fix structural metrics rate by aggregation #5204 (@zalegrala)
• [BUGFIX] TraceQL Metrics: right exemplars for histogram and quantiles #5145 (@ruslan-mikhailov)

Breaking Changes

• [CHANGE] BREAKING CHANGE Change default http-listen-port from 80 to 3200 #4960 (@martialblog)
• [CHANGE] BREAKING CHANGE Upgrade OTEL Collector to v0.122.1 #4893 (@javiermolinar)
  The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver
• [CHANGE] BREAKING CHANGE Convert SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it's more performant. #4748 (@carles-grafana)
• [CHANGE] BREAKING CHANGE Remove tempo serverless #4599 (@electron0zero)
  Following config options are no longer valid, please remove them if you are using these in your tempo config:

  querier:
      search:
          prefer_self: <int>
          external_hedge_requests_at: <duration>
          external_hedge_requests_up_to:  <duration>
          external_backend: <string>
          google_cloud_run: <string>
          external_endpoints: <array>
  

  Tempo serverless related metric tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total and tempo_feature_enabled are being removed.
• [CHANGE] BREAKING CHANGE Removed internal_error as a reason from tempo_discarded_spans_total. #4554 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Enforce max attribute size at event, link, and instrumentation scope. Make config per-tenant.
  Renamed max_span_attr_byte to max_attribute_bytes #4633 (@ie-pham)
• [CHANGE] BREAKING CHANGE Removed otel jaeger exporter. #4926 (@javiermolinar)

Changes

• [CHANGE] Update query range error message #4929 (@joey-grafana)
• [CHANGE] Improve rate limit error message when traces size exceeds rate limit #4986 (@electron0zero)
• [CHANGE] Tempo Query: replace opentracing-contrib/go-grpc by otelgrpc. #4958 (@javiermolinar)
• [CHANGE] Improve parquet readers io.ReaderAt compatibility #4963 (@joe-elliott)
• [CHANGE] Finish polling current tenants on poller shutdown #4897 (@zalegrala)
• [CHANGE] Set querier default level to INFO #4943 (@javiermolinar)
• [CHANGE] Change retention to honor compactor disablement #5044 (@zalegrala)
• [CHANGE] Continue on error in tempo-cli rewrite-blocks #5043 (@zalegrala)
• [CHANGE] Update jsonnet memcached and exporter image versions #5056 (@zalegrala)
• [CHANGE] Enforce max series in response for metrics queries with new query-frontend configuration #4525 (@ie-pham)
• [CHANGE] Update to go 1.24.3 #5110 (@stoewer)
• [CHANGE] Update to go 1.24.2 #5057 #5082 (@carsontham)
• [CHANGE] Update to go 1.24.1 #4704 (@ruslan-mikhailov) #4793 (@javiermolinar)

Features

• [FEATURE] Added most_recent=true query hint to TraceQL to return most recent results. #4238 (@joe-elliott)
• [FEATURE] TraceQL metrics: sum_over_time #4786 (@javiermolinar)
• [FEATURE] Add support for topk and bottomk functions for TraceQL metrics #4646 (@electron0zero)
• [FEATURE] TraceQL: add support for querying by parent span id #4692 (@ie-pham)

Enhancements

• [ENHANCEMENT] Add throughput SLO and metrics for the TraceByID endpoint. #4668 (@carles-grafana)
  configurable via the throughput_bytes_slo field, and it will populate op="traces" label in slo and throughput metrics.
• [ENHANCEMENT] Add ability to add artificial delay to push requests #4716 #4899 #5035 (@yvrhdn, @mapno)
• [ENHANCEMENT] tempo-vulture now generates spans with a parent, instead of only root spans #5154 (@carles-grafana)
• [ENHANCEMENT] Add default mutex and blocking values. #4979 (@mattdurham)
• [ENHANCEMENT] Improve Tempo build options #4755 (@stoewer)
• [ENHANCEMENT] Rewrite traces using rebatching #4690 (@stoewer @joe-elliott)
• [ENHANCEMENT] Reorder span iterators #4754 (@stoewer)
• [ENHANCEMENT] Update minio to version #4341 (@javiermolinar)
• [ENHANCEMENT] Prevent queries in the ingester from blocking flushing traces to disk and memory spikes. #4483 (@joe-elliott)
• [ENHANCEMENT] Update tempo operational dashboard for new block-builder and v2 traces api #4559 (@mdisibio)
• [ENHANCEMENT] Improve metrics-generator performance and stability by applying queue back pressure and concurrency #4721 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by flushing blocks concurrently #4565 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance #4596 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by not using WAL stage #4647 #4671 (@mdisibio)
• [ENHANCEMENT] Export new tempo_ingest_group_partition_lag metric from block-builders and metrics-generators #4571 (@mdisibio)
• [ENHANCEMENT] Overall iterator performance improvement by using max definition level to ignore parts of the RowNumber while nexting. #4753 (@joe-elliott)
• [ENHANCEMENT] Use distroless base container images for improved security #4556 (@carles-grafana)
• [ENHANCEMENT] Rhythm: add block builder to resources dashboard #4556 (@javiermolinar)
• [ENHANCEMENT] Upgrade prometheus to version 3.1.0 #4805 (@javiermolinar)
• [ENHANCEMENT] Update dskit to latest version #4681 (@javiermolinar) #4865 (@javiermolinar)
• [ENHANCEMENT] Increase query-frontend default batch size #4844 (@javiermolinar)
• [ENHANCEMENT] Improve TraceQL perf by reverting EqualRowNumber to an inlineable function.#4705 (@joe-elliott)
• [ENHANCEMENT] Rhythm: Implement MaxBytesPerCycle #4835 (@javiermolinar)
• [ENHANCEMENT] Rhythm: fair partition consumption in blockbuilders #4655 (@javiermolinar)
• [ENHANCEMENT] Rhythm: retry on commit error #4874 (@javiermolinar)
• [ENHANCEMENT] Skip creating one span-traces for every pushed spans in metrics generator #4844 (@javiermolinar)
• [ENHANCEMENT] Improve Tempo / Writes dashboard by adding a kafka panel #4947 (@javiermolinar)
• [ENHANCEMENT] Improve memcached memory usage by pooling buffers #4970 (@joe-elliott)
• [ENHANCEMENT] metrics-generator: allow skipping localblocks and consuming from a different source of data #4686 (@flxbk)
• [ENHANCEMENT] compactor: restore dedicated columns logging for completed blocks #4832 (@edgarkz)
• [ENHANCEMENT] Compactor: pooling changes to reduce memory usage #4985 (@mdisibio)
• [ENHANCEMENT] distributor: add IPv6 support #4840 (@gjacquet)
• [ENHANCEMENT] Support TraceQL Metrics checks in Vulture #4886 (@ruslan-mikhailov)
• [ENHANCEMENT] Add memcached to the resources dashboard #5049 (@javiermolinar)
• [ENHANCEMENT] Include partition owned metric for blockbuilders #5042 (@javiermolinar)
• [ENHANCEMENT] Query-frontend: logs add msg to the log line #4975 (@jmichalek132)
• [ENHANCEMENT] Host Info Processor: track host identifying resource attribute in metric #5152 (@rlankfo)
• [ENHANCEMENT] Vulture checks recent traces #5157 (@ruslan-mikhailov)
• [ENHANCEMENT] Add jitter in backendworker to avoid thundering herd from workers [#515...

Bugfixes

• [BUGFIX] Fix rare panic that occurred when a querier modified results from ingesters/generators while they were being marshalled to proto. #4790 (@joe-elliott)
  This bug also impacted query correctness on recent trace data by returning incomplete results before they were ready.

Thanks to @pavleec for filing an issue bringing the query correctness element of this bug to our attention as well as testing a fix.

Changes

• [CHANGE] Default to snappy compression for all gRPC communications internal to Tempo. We feel this is a nice balance of resource usage and network traffic. For a discussion on alternatives see https://github.com/grafana/tempo/discussions/4683. #4696 (@joe-elliott)

Thanks to @edgarkz for championing this issue and providing details of the various compression settings on their install.

Deprecations

• Tempo serverless features are now deprecated and will be removed in an upcoming release #4017 @electron0zero

Breaking Changes

• Added maximum spans per span set to prevent queries from overwhelming read path. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger
• Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
• Removed querier_forget_delay setting from the frontend. This configuration option did nothing. #3996 (@joe-elliott)
• Use Prometheus fast regexp for TraceQL regular expression matchers. #4329 (@joe-elliott)
  All regular expression matchers will now be fully anchored. span.foo =~ "bar" will now be evaluated as span.foo =~ "^bar$"

Changes

• [CHANGE] Disable gRPC compression in the querier and distributor for performance reasons #4429 (@carles-grafana)
  If you would like to re-enable it, we recommend 'snappy'. Use the following settings:

ingester_client:
    grpc_client_config:
        grpc_compression: "snappy"
metrics_generator_client:
    grpc_client_config:
        grpc_compression: "snappy"
querier:
    frontend_worker:
        grpc_client_config:
            grpc_compression: "snappy"

• [CHANGE] slo: include request cancellations within SLO [#4355] (#4355) (@electron0zero)
  request cancellations are exposed under result label in tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total with completed or canceled values to differentiate between completed and canceled requests.
• [CHANGE] update default config values to better align with production workloads #4340 (@electron0zero)
• [CHANGE] tempo-cli: add support for /api/v2/traces endpoint #4127 (@electron0zero)
  BREAKING CHANGE The tempo-cli now uses the /api/v2/traces endpoint by default,
  please use --v1 flag to use /api/traces endpoint, which was the default in previous versions.
• [CHANGE] TraceByID: don't allow concurrent_shards greater than query_shards. #4074 (@electron0zero)
• [CHANGE] BREAKING CHANGE The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• [CHANGE] BREAKING CHANGE Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger.
• [CHANGE] No longer send the final diff in GRPC streaming. Instead we rely on the streamed intermediate results. #4062 (@joe-elliott)
• [CHANGE] Update Go to 1.23.3 #4146 #4147 #4380 (@javiermolinar @mdisibio)
• [CHANGE] Return 422 for TRACE_TOO_LARGE queries #4160 (@zalegrala)
• [CHANGE] Tighten file permissions #4251 (@zalegrala)
• [CHANGE] Drop max live traces log message and rate limit trace too large. #4418 (@joe-elliott)
• [CHANGE] Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  BREAKING CHANGE After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

Features

• [FEATURE] tempo-cli: support dropping multiple traces in a single operation #4266 (@ndk)
• [FEATURE] Discarded span logging log_discarded_spans #3957 (@dastrobu)
• [FEATURE] TraceQL support for instrumentation scope #3967 (@ie-pham)
• [FEATURE] Export cost attribution usage metrics from distributor #4162 (@mdisibio)
• [FEATURE] TraceQL metrics: avg_over_time #4073 (@javiermolinar)
• [FEATURE] TraceQL metrics queries: add min_over_time #3975 (@javiermolinar)
• [FEATURE] TraceQL metrics queries: add max_over_time #4065 (@javiermolinar)
• [FEATURE] Limit tags and tag values search #4320 (@javiermolinar)

Enhancements

• [ENHANCEMENT] TraceQL: Add range condition for byte predicates #4198 (@ie-pham)
• [ENHANCEMENT] Add throughput and SLO metrics in the tags and tag values endpoints #4148 (@electron0zero)
• [ENHANCEMENT] BREAKING CHANGE Add maximum spans per span set. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• [ENHANCEMENT] Add query-frontend limit for max length of query expression #4397 (@electron0zero)
• [ENHANCEMENT] distributor: return trace id length when it is invalid #4407 (@carles-grafana)
• [ENHANCEMENT] Changed log level from INFO to DEBUG for the TempoDB Find operation using traceId to reduce excessive/unwanted logs in log search. #4179 (@Aki0x137)
• [ENHANCEMENT] Pushdown collection of results from generators in the querier #4119 (@electron0zero)
• [ENHANCEMENT] The span multiplier now also sources its value from the resource attributes. #4210 (@JunhoeKim)
• [ENHANCEMENT] TraceQL: Attribute iterators collect matched array values #3867 (@electron0zero, @stoewer)
• [ENHANCEMENT] Allow returning partial traces that exceed the MaxBytes limit for V2 #3941 (@javiermolinar)
• [ENHANCEMENT] Added new middleware to validate request query values #3993 (@javiermolinar)
• [ENHANCEMENT] Prevent massive allocations in the frontend if there is not sufficient pressure from the query pipeline. #3996 (@joe-elliott)
  BREAKING CHANGE Removed querier_forget_delay setting from the frontend. This configuration option did nothing.
• [ENHANCEMENT] Update metrics-generator config in Tempo distributed docker compose example to serve TraceQL metrics #4003 (@javiermolinar)
• [ENHANCEMENT] Reduce allocs related to marshalling dedicated columns repeatedly in the query frontend. #4007 (@joe-elliott)
• [ENHANCEMENT] Improve performance of TraceQL queries #4114 (@mdisibio)
• [ENHANCEMENT] Improve performance of TraceQL queries #4163 (@mdisibio)
• [ENHANCEMENT] Improve performance of some TraceQL queries using select() operation #4438 (@mdisibio)
• [ENHANCEMENT] Reduce memory usage of classic histograms in the span-metrics and service-graphs processors #4232 (@mdisibio)
• [ENHANCEMENT] ...

Deprecations

• Tempo serverless features are now deprecated and will be removed in an upcoming release #4017 @electron0zero

Breaking Changes

• Added maximum spans per span set to prevent queries from overwhelming read path. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger
• Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
• Removed querier_forget_delay setting from the frontend. This configuration option did nothing. #3996 (@joe-elliott)
• Use Prometheus fast regexp for TraceQL regular expression matchers. #4329 (@joe-elliott)
  All regular expression matchers will now be fully anchored. span.foo =~ "bar" will now be evaluated as span.foo =~ "^bar$"

Changes

• [CHANGE] Disable gRPC compression in the querier and distributor for performance reasons #4429 (@carles-grafana)
  If you would like to re-enable it, we recommend 'snappy'. Use the following settings:

ingester_client:
    grpc_client_config:
        grpc_compression: "snappy"
metrics_generator_client:
    grpc_client_config:
        grpc_compression: "snappy"
querier:
    frontend_worker:
        grpc_client_config:
            grpc_compression: "snappy"

• [CHANGE] slo: include request cancellations within SLO [#4355] (#4355) (@electron0zero)
  request cancellations are exposed under result label in tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total with completed or canceled values to differentiate between completed and canceled requests.
• [CHANGE] update default config values to better align with production workloads #4340 (@electron0zero)
• [CHANGE] tempo-cli: add support for /api/v2/traces endpoint #4127 (@electron0zero)
  BREAKING CHANGE The tempo-cli now uses the /api/v2/traces endpoint by default,
  please use --v1 flag to use /api/traces endpoint, which was the default in previous versions.
• [CHANGE] TraceByID: don't allow concurrent_shards greater than query_shards. #4074 (@electron0zero)
• [CHANGE] BREAKING CHANGE The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
• [CHANGE] BREAKING CHANGE Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr)
  To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger.
• [CHANGE] No longer send the final diff in GRPC streaming. Instead we rely on the streamed intermediate results. #4062 (@joe-elliott)
• [CHANGE] Update Go to 1.23.3 #4146 #4147 #4380 (@javiermolinar @mdisibio)
• [CHANGE] Return 422 for TRACE_TOO_LARGE queries #4160 (@zalegrala)
• [CHANGE] Tighten file permissions #4251 (@zalegrala)
• [CHANGE] Drop max live traces log message and rate limit trace too large. #4418 (@joe-elliott)
• [CHANGE] Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn)
  BREAKING CHANGE After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0.
  Due to this, Tempo installations running inside Docker have to update the address they listen.
  For more details on this change, see #4465
  For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

Features

• [FEATURE] tempo-cli: support dropping multiple traces in a single operation #4266 (@ndk)
• [FEATURE] Discarded span logging log_discarded_spans #3957 (@dastrobu)
• [FEATURE] TraceQL support for instrumentation scope #3967 (@ie-pham)
• [FEATURE] Export cost attribution usage metrics from distributor #4162 (@mdisibio)
• [FEATURE] TraceQL metrics: avg_over_time #4073 (@javiermolinar)
• [FEATURE] TraceQL metrics queries: add min_over_time #3975 (@javiermolinar)
• [FEATURE] TraceQL metrics queries: add max_over_time #4065 (@javiermolinar)
• [FEATURE] Limit tags and tag values search #4320 (@javiermolinar)

Enhancements

• [ENHANCEMENT] TraceQL: Add range condition for byte predicates #4198 (@ie-pham)
• [ENHANCEMENT] Add throughput and SLO metrics in the tags and tag values endpoints #4148 (@electron0zero)
• [ENHANCEMENT] BREAKING CHANGE Add maximum spans per span set. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
• [ENHANCEMENT] Add query-frontend limit for max length of query expression #4397 (@electron0zero)
• [ENHANCEMENT] distributor: return trace id length when it is invalid #4407 (@carles-grafana)
• [ENHANCEMENT] Changed log level from INFO to DEBUG for the TempoDB Find operation using traceId to reduce excessive/unwanted logs in log search. #4179 (@Aki0x137)
• [ENHANCEMENT] Pushdown collection of results from generators in the querier #4119 (@electron0zero)
• [ENHANCEMENT] The span multiplier now also sources its value from the resource attributes. #4210 (@JunhoeKim)
• [ENHANCEMENT] TraceQL: Attribute iterators collect matched array values #3867 (@electron0zero, @stoewer)
• [ENHANCEMENT] Allow returning partial traces that exceed the MaxBytes limit for V2 #3941 (@javiermolinar)
• [ENHANCEMENT] Added new middleware to validate request query values #3993 (@javiermolinar)
• [ENHANCEMENT] Prevent massive allocations in the frontend if there is not sufficient pressure from the query pipeline. #3996 (@joe-elliott)
  BREAKING CHANGE Removed querier_forget_delay setting from the frontend. This configuration option did nothing.
• [ENHANCEMENT] Update metrics-generator config in Tempo distributed docker compose example to serve TraceQL metrics #4003 (@javiermolinar)
• [ENHANCEMENT] Reduce allocs related to marshalling dedicated columns repeatedly in the query frontend. #4007 (@joe-elliott)
• [ENHANCEMENT] Improve performance of TraceQL queries #4114 (@mdisibio)
• [ENHANCEMENT] Improve performance of TraceQL queries #4163 (@mdisibio)
• [ENHANCEMENT] Improve performance of some TraceQL queries using select() operation #4438 (@mdisibio)
• [ENHANCEMENT] Reduce memory usage of classic histograms in the span-metrics and service-graphs processors #4232 (@mdisibio)
• [ENHANCEMENT] ...

Bugfixes

• [BUGFIX] Bring back application-json content-type header. #4123 (@javiermolinar)

Changes

• [CHANGE] BREAKING CHANGE tempo-query is no longer a jaeger instance with grpcPlugin. Its now a standalone server. Serving a grpc api for jaeger on 0.0.0.0:7777 by default. #3840 (@frzifus)

Enhancements

• [ENHANCEMENT] Register gRPC health server to tempo-query #4178 (@frzifus)
• [ENHANCEMENT] Support Tempo on IBM s390x #4175 (@pavolloffay)
• [ENHANCEMENT] tempo-query: separate tls settings for server and client #4177 (@frzifus)
• [ENHANCEMENT] Speedup tempo-query trace search by allowing parallel queries #4159 (@pavolloffay)

Breaking Changes

• autocomplete_filtering_enabled feature flag has been removed and always enabled.
• use_v2_sdk in the Azure backend has been removed. Azure v2 sdk has been promoted to GA/dfeault.
• Removed completedfilepath and blocksfilepath from the wal config.
• TraceQL metrics now bases historical requests off of a set of RF1 blocks written to the backend by the metrics-generators. Set flush_to_storage to true to run metrics queries on historical data.

  metrics_generator:
    processor:
      local_blocks:
        flush_to_storage: true
  

Changes

• [CHANGE] BREAKING CHANGE Remove autocomplete_filtering_enabled feature flag #3729 (@mapno)
• [CHANGE] Bump opentelemetry-collector to 0.102.1 #3784 (@debasishbsws)
• [CHANGE] Bump Jaeger query docker image to 1.57.0 #3652 (@iblancasa)
• [CHANGE] Update Go to 1.22.4 #3757 #3793 (@joe-elliott, @mapno)
• [CHANGE] Make vParquet4 the default block encoding #3810 (@ie-pham)
• [CHANGE] Azure v2 backend becomes the only and primary Azure backend #3875 (@zalegrala)
  BREAKING CHANGE The use_v2_sdk configuration option has been removed.

Features

• [FEATURE] TraceQL support for link scope and link:traceID and link:spanID #3741 (@stoewer)
• [FEATURE] TraceQL support for link attribute querying #3814 (@ie-pham)
• [FEATURE] TraceQL support for event scope and event:name intrinsic #3708 (@stoewer)
• [FEATURE] TraecQL support for event attributes #3708 (@ie-pham)
• [FEATURE] TraceQL support for event:timeSinceStart #3908 (@ie-pham)
• [FEATURE] Autocomplete support for events and links #3846 (@ie-pham)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno)
• [FEATURE] Add new compare() metrics function #3695 (@mdisibio)
• [FEATURE] Add new api /api/metrics/query for instant metrics queries #3859 (@mdisibio)
• [FEATURE] Add a q parameter to /api/v2/search/tags for tag name filtering #3822 (@joe-elliott)
• [FEATURE] Add exemplars to TraceQL metrics #3824 (@mapno)
• [FEATURE] Added new middleware to block matching urls #3963 (@javiermolinar)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno) #3995 (@mdisibio)
• [FEATURE] Add support for generating native histograms from metrics-generator #3789 (@zalegrala, @kvrhdn)

Enhancements

• [ENHANCEMENT] Implement arrays for traceql.Static with reused fields #3827 (@stoewer)
• [ENHANCEMENT] Tag value lookup use protobuf internally for improved latency #3731 (@mdisibio)
• [ENHANCEMENT] TraceQL metrics queries use protobuf internally for improved latency #3745 (@mdisibio)
• [ENHANCEMENT] TraceQL search and other endpoints use protobuf internally for improved latency and resource usage #3944 (@mdisibio)
• [ENHANCEMENT] Add local disk caching of metrics queries in local-blocks processor #3799 (@mdisibio)
• [ENHANCEMENT] Improve use of OTEL semantic conventions on the service graph #3711 (@zalegrala)
• [ENHANCEMENT] Performance improvement for rate() by () queries #3719 (@mapno)
• [ENHANCEMENT] Performance improvement for queries using trace-level intrinsics #3920 (@mdisibio)
• [ENHANCEMENT] Use multiple goroutines to unmarshal responses in parallel in the query frontend. #3713 (@joe-elliott)
• [ENHANCEMENT] Protect ingesters from panics by adding defer/recover to all read path methods. #3790 (@joe-elliott)
• [ENHANCEMENT] Added a boolean flag to enable or disable dualstack mode on Storage block config for S3 #3721 (@sid-jar, @mapno)
• [ENHANCEMENT] Add caching to query range queries #3796 (@mapno)
• [ENHANCEMENT] Only stream diffs on metrics queries #3808 (@joe-elliott)
• [ENHANCEMENT] Add data quality metric to measure traces without a root #3812 (@mapno)
• [ENHANCEMENT] Added an example for running Tempo vulture #3829 (@javiermolinar)
• [ENHANCEMENT] Add a new helper method to allow debugging e2e tests #3836 (@javiermolinar)
• [ENHANCEMENT] Refactor Tempo Vulture to reduce code complexity #3850 (@javiermolinar)
• [ENHANCEMENT] Self document makefile #3844 (@javiermolinar)
• [ENHANCEMENT] Added a Tempo CLI command to drop traces by id by rewriting blocks. #3856 (@joe-elliott)
• [ENHANCEMENT] Mixin, make recording rule range interval configurable and increase range interval in alert to support scrape interval of 1 minute #3851 (@jmichalek132)
• [ENHANCEMENT] Add vParquet4 support to the tempo-cli analyse blocks command #3868 (@stoewer)
• [ENHANCEMENT] Improve trace id lookup from Tempo Vulture by selecting a date range #3874 (@javiermolinar)
• [ENHANCEMENT] Add native histograms for internal metrics#3870 (@zalegrala)
• [ENHANCEMENT] Expose availability-zone as a cli flag in ingester #3881
• [ENHANCEMENT] Rename batches property of Trace to ResourceSpans to be OTEL compatible #3895
• [ENHANCEMENT] Reduce memory consumption of query-frontend#3888 (@joe-elliott)
• [ENHANCEMENT] Reduce log level verbosity for e2e tests#3900 (@javiermolinar)
• [ENHANCEMENT] Added new Traces api V2#3912 (@javiermolinar)
• [ENHANCEMENT] Update to the latest dskit #3915 (@andreasgerstmayr)
• [ENHANCEMENT] Reduce allocs building queriers sharded requests #3932 (@javiermolinar)
• [ENHANCEMENT] Allow compaction disablement per-tenant #3965 (@zalegrala)
• [ENHANCEMENT] Implement polling tenants concurrently #3647 (@zalegrala)
• [ENHANCEMENT] Reduce allocs of caching middleware #3976 (@joe-elliott)
• [ENHANCEMENT] BlockMeta improvements to reduce the size #3950 #3951 #3952(@zalegrala)
• [ENHANCEMENT] Add bytes and spans received to usage stats #3983 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix panic in certain metrics queries using rate() with by #3847 (@stoewer)
• [BUGFIX] Fix double appending the primary iterator on second pass with event iterator #3903 (@ie-pham)
• [BUGFIX] Fix metrics queries when grouping by attributes that may not exist #3734 (@mdisibio)
• [BUGFIX] Fix frontend parsing error on cached responses #3759 (@mdisibio)
• [BUGFIX] Fix autocomplete of a query using scoped instrinsics #3865 (@mdisibio)
• [BUGFIX] Fix metrics query histograms and quantiles on traceDuration #3879 (@mdisibio)
• [BUGFIX] Fix divide by 0 bug in query frontend exemplar calculations #3936 (@mdisibio)
• [BUGFIX] max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit #3618 (@kvrhdn)
• [BUGFIX] Fix http connection reuse on GCP and AWS by reading io.EOF through the http body. #3760 (@bmteller)
• [BUGFIX] Improved handling of complete blocks in localblocks processor after enabling flusing #3805 (@mapno)
• [BUGFIX] ...