This release contains 590 PRs from 62 authors, including new contributors Andrew Hall, Frank Kloeker, Gregor Zeitlinger, Ilia Lazebnik, Julien, Maruth Goyal, Matt Clegg, Mattia Barbon, Mohijeet, Oleg Vorobev, Oliver Herrmann, Phil Kates, Pranshu Raj, Rob B, Timon Engelke, Vadim Stepanov, dannyc-grafana, renovate-sh-app[bot], unkls ben. Thank you!

Grafana Mimir version 3.0.0 release notes

Grafana Labs is excited to announce version 3.0 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Grafana Mimir version 3.0 includes the following key features and enhancements.

Ingest storage architecture

Mimir 3.0 introduces ingest storage architecture, a next-generation architecture that separates the read and write paths using a Kafka-based ingest storage layer. In previous versions, the ingester handled both ingestion and querying, which could cause contention under heavy query loads. By introducing Kafka as an asynchronous buffer between ingestion and query, reads and writes now scale independently. This design improves ingestion stability, query performance, and operational flexibility for large deployments.

Note that in Grafana Mimir version 3.0, classic architecture is still supported.

Mimir Query Engine (MQE)

The Mimir Query Engine (MQE) is now the default query engine for all queriers and query-frontends. MQE is fully PromQL-compatible and processes data in a streaming fashion, reducing peak memory usage by up to 92% compared to the Prometheus engine. You can expect faster, more stable query performance and lower resource consumption under load.

Additional improvements

Grafana Mimir 3.0 also includes:

• Optimizations to MQE for histogram queries and query rewrites
• Improvements to query planning in frontends and queriers
• General stability and performance updates across core components

Important changes

Grafana Mimir 3.0 introduces several updates that change default behavior and configuration. Review these changes before upgrading:

• Ingest storage architecture: Grafana Mimir now uses a Kafka-based ingest storage layer that separates reads and writes. This change improves performance by altering data flow between components.
• MQE as the default engine: MQE replaces the Prometheus engine as the default query engine. To continue using the Prometheus engine, set -querier.query-engine=prometheus.
• Removal of experimental features: Some deprecated and experimental flags, including the read-write deployment mode, have been removed.
• Configuration updates: Some deprecated CLI flags and configuration options have been removed or renamed.

Experimental features

Grafana Mimir 3.0 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol
• PromQL duration expressions that allow simple arithmetic in range or offset values
• Cluster validation for HTTP requests (-server.cluster-validation.* flags)
• Native OTLP delta metric ingestion (-distributor.otel-native-delta-ingestion)

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir Helm chart is released independently. Refer to the Grafana Mimir Helm chart documentation.

Changelog

3.0.0

Grafana Mimir

• [CHANGE] Build: Include updated Mozilla CA bundle from Debian Testing. #12247
• [CHANGE] Query-frontend: Add support for UTF-8 label and metric names in /api/v1/cardinality/{label_values|label_values|active_series} endpoints. #11848.
• [CHANGE] Querier: Add support for UTF-8 label and metric names in label_join, label_replace and count_values PromQL functions. #11848.
• [CHANGE] Remove support for Redis as a cache backend. #12163
• [CHANGE] Memcached: Remove experimental -<prefix>.memcached.addresses-provider flag to use alternate DNS service discovery backends. The more reliable backend introduced in 2.16.0 (#10895) is now the default. As a result of this change, DNS-based cache service discovery no longer supports search domains. #12175 #12385
• [CHANGE] Query-frontend: Remove the CLI flag -query-frontend.downstream-url and corresponding YAML configuration and the ability to use the query-frontend to proxy arbitrary Prometheus backends. #12191 #12517
• [CHANGE] Query-frontend: Remove experimental instant query splitting feature. #12267
• [CHANGE] Query-frontend, querier: Replace query-frontend.prune-queries flag with querier.mimir-query-engine.enable-prune-toggles as pruning middleware has been moved into MQE. #12303 #12375
• [CHANGE] Distributor: Remove deprecated global HA tracker timeout configuration flags. #12321
• [CHANGE] Query-frontend: Use the Mimir Query Engine (MQE) by default. #12361
• [CHANGE] Query-frontend: Remove the CLI flags -querier.frontend-address, -querier.max-outstanding-requests-per-tenant, and -query-frontend.querier-forget-delay and corresponding YAML configurations. This is part of a change that makes the query-scheduler a required component. This removes the ability to run the query-frontend with an embedded query-scheduler. Instead, you must run a dedicated query-scheduler component. #12200
• [CHANGE] Ingester: Remove deprecated -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option. #12615
• [CHANGE] Querier: Require non-zero values for -querier.streaming-chunks-per-ingester-buffer-size and -querier.streaming-chunks-per-store-gateway-buffer-size CLI flags and corresponding YAML configurations. This is part of a change that makes streaming required between queriers, ingesters, and store-gateways. Streaming has been the default since Mimir 2.14. #12790 #12818 #12897 #12929 #12973
• [CHANGE] Remove support for the experimental read-write deployment mode. #12584
• [CHANGE] Store-gateway: Update default value of -store-gateway.dynamic-replication.multiple to 5 to increase replication of recent blocks. #12433
• [CHANGE] Cost attribution: Reduce the default maximum per-user cardinality of cost attribution labels to 2000. #12625
• [CHANGE] Querier, query-frontend: Add _total suffix to cortex_mimir_query_engine_common_subexpression_elimination_duplication_nodes_introduced, cortex_mimir_query_engine_common_subexpression_elimination_selectors_eliminated and cortex_mimir_query_engine_common_subexpression_elimination_selectors_inspected metric names. #12636
• [CHANGE] Distributor: Remove the experimental setting service_overload_status_code_on_rate_limit_enabled which used an HTTP 529 error (non-standard) instead of HTTP 429 for rate limiting. #13012
• [CHANGE] Alertmanager: Change the severity for InitialSyncFailed from 'critical' to 'warning'. #12824
• [CHANGE] Ingester: Renamed experimental reactive limiter options. #12773
• [CHANGE] Distributor: gRPC errors with the mimirpb.ERROR_CAUSE_INSTANCE_LIMIT cause are now mapped to codes.Unavailable and http.StatusServiceUnavailable instead of codes.Internal and http.StatusInternalServerError. #13003 #13032
• [CHANGE] Admin: use relative links instead of absolute ones in the administration web UI. #13034
• [CHANGE] Distributor: Use memberlist by default for the HA tracker. #12998
• [CHANGE] Block-builder: Remove cortex_blockbuilder_process_partition_duration_seconds metric and related dashboard panels. #12631
• [FEATURE] Ingester: Expose the number of active series ingested via OTLP as cortex_ingester_active_otlp_series. #12678
• [FEATURE] Distributor, ruler: Add experimental -validation.name-validation-scheme flag to specify the validation scheme for metric and label names. #12215
• [FEATURE] Ruler: Add support to use a Prometheus-compatible HTTP endpoint for remote rule evaluation. See remote evaluation mode for more details. This feature can be used to federate data from multiple Mimir instances. #11415 #11833
• [FEATURE] Distributor: Add experimental -distributor.otel-translation-strategy flag to support configuring the metric and label name translation strategy in the OTLP endpoint. #12284 #12306 #12369
• [FEATURE] Query-frontend: Add query-frontend.rewrite-propagate-matchers flag that enables a new MQE AST optimization pass that copies relevant label matchers across binary operations. #12304
• [FEATURE] Query-frontend: Add query-frontend.rewrite-histogram-queries flag that enables a new MQE AST optimization pass that rewrites histogram queries for a more efficient order of execution. #12305
• [FEATURE] Query-frontend: Support delayed name removal (Prometheus experimental feature) in MQE. #12509
• [FEATURE] Usage-tracker: Introduce a new experimental service to enforce active series limits before Kafka ingestion. #12358 #12895 #12940 #12942 #12970 #13085
• [FEATURE] Ingester: Add experimental -include-tenant-id-in-profile-labels flag to include tenant ID in pprof profiling labels for sampled traces. Currently only supported by the ingester. This can help debug performance issues for specific tenants. #12404
• [FEATURE] Alertmanager: Add experimental -alertmanager.storage.state-read-timeout flag to configure the timeout for reading the Alertmanager state (notification log, silences) from object storage during the initial sync. #12425
• [FEATURE] Ingester: Add experimental -blocks-storage.tsdb.index-lookup-planning.* flags to configure use of a cost-based index lookup planner. This sho...

This release contains 11 PRs from 6 authors. Thank you!

Changelog

2.17.2

Grafana Mimir

• [BUGFIX] Add a missing attribute to the list of default promoted OTel resource attributes in the docs: deployment.environment. #12181
• [BUGFIX] Ingest: Fix memory pool poisoning in Remote-Write 2.0/OTLP by not cleaning created timestamp field before returning time series to the memory pool. #12735
• [BUGFIX] Distributor: Fix error when native histograms bucket limit is set then no NHCB passes validation. #12746
• [BUGFIX] Update Docker base images for tools from alpine:3.22.1 to alpine:3.22.2 to address CVE-2025-9230, CVE-2025-9231, CVE-2025-2025-9232. #12993
• [BUGFIX] Memcached: Ignore invalid responses when discovering cache servers using dnssrv+ or dnssrvnoa+ service discovery prefixes. #13206

Tools

• [ENHANCEMENT] Base mimirtool, metaconvert, copyblocks, and query-tee images on distroless/static-debian12. #13014

All changes in this release: mimir-2.17.1...mimir-2.17.2

This release contains 3 PRs from 2 authors. Thank you!

Changelog

3.0.0-rc.2

Grafana Mimir

• [BUGFIX] Memcached: Ignore invalid responses when discovering cache servers using dnssrv+ or dnssrvnoa+ service discovery prefixes. #13203

All changes in this release: mimir-3.0.0-rc.1...mimir-3.0.0-rc.2

This release contains 586 PRs from 62 authors, including new contributors Andrew Hall, Frank Kloeker, Gregor Zeitlinger, Ilia Lazebnik, Julien, Maruth Goyal, Matt Clegg, Mattia Barbon, Mohijeet, Oleg Vorobev, Oliver Herrmann, Phil Kates, Pranshu Raj, Rob B, Timon Engelke, Vadim Stepanov, dannyc-grafana, unkls ben. Thank you!

Grafana Mimir version 3.0.0-rc.1 release notes

Grafana Labs is excited to announce version 3.0 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Grafana Mimir version 3.0 includes the following key features and enhancements.

Ingest storage architecture

Mimir 3.0 introduces ingest storage architecture, a next-generation architecture that separates the read and write paths using a Kafka-based ingest storage layer. In previous versions, the ingester handled both ingestion and querying, which could cause contention under heavy query loads. By introducing Kafka as an asynchronous buffer between ingestion and query, reads and writes now scale independently. This design improves ingestion stability, query performance, and operational flexibility for large deployments.

Note that in Grafana Mimir version 3.0, classic architecture is still supported.

Mimir Query Engine (MQE)

The Mimir Query Engine (MQE) is now the default query engine for all queriers and query-frontends. MQE is fully PromQL-compatible and processes data in a streaming fashion, reducing peak memory usage by up to 92% compared to the Prometheus engine. You can expect faster, more stable query performance and lower resource consumption under load.

Additional improvements

Grafana Mimir 3.0 also includes:

• Optimizations to MQE for histogram queries and query rewrites
• Improvements to query planning in frontends and queriers
• General stability and performance updates across core components

Important changes

Grafana Mimir 3.0 introduces several updates that change default behavior and configuration. Review these changes before upgrading:

• Ingest storage architecture: Grafana Mimir now uses a Kafka-based ingest storage layer that separates reads and writes. This change improves performance by altering data flow between components.
• MQE as the default engine: MQE replaces the Prometheus engine as the default query engine. To continue using the Prometheus engine, set -querier.query-engine=prometheus.
• Removal of experimental features: Some deprecated and experimental flags, including the read-write deployment mode, have been removed.
• Configuration updates: Some deprecated CLI flags and configuration options have been removed or renamed.

Experimental features

Grafana Mimir 3.0 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol
• PromQL duration expressions that allow simple arithmetic in range or offset values
• Cluster validation for HTTP requests (-server.cluster-validation.* flags)
• Native OTLP delta metric ingestion (-distributor.otel-native-delta-ingestion)

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir Helm chart is released independently. Refer to the Grafana Mimir Helm chart documentation.

Changelog

3.0.0-rc.1

Jsonnet

• [CHANGE] Mimir-continuous-test: Use mimir -target=continuous-test instead of standalone binary/image. #13097

Tools

• [CHANGE] Mimir-continuous-test: Remove standalone binary and image. #13097

3.0.0-rc.0

Grafana Mimir

• [CHANGE] Build: Include updated Mozilla CA bundle from Debian Testing. #12247
• [CHANGE] Query-frontend: Add support for UTF-8 label and metric names in /api/v1/cardinality/{label_values|label_values|active_series} endpoints. #11848.
• [CHANGE] Querier: Add support for UTF-8 label and metric names in label_join, label_replace and count_values PromQL functions. #11848.
• [CHANGE] Remove support for Redis as a cache backend. #12163
• [CHANGE] Memcached: Remove experimental -<prefix>.memcached.addresses-provider flag to use alternate DNS service discovery backends. The more reliable backend introduced in 2.16.0 (#10895) is now the default. As a result of this change, DNS-based cache service discovery no longer supports search domains. #12175 #12385
• [CHANGE] Query-frontend: Remove the CLI flag -query-frontend.downstream-url and corresponding YAML configuration and the ability to use the query-frontend to proxy arbitrary Prometheus backends. #12191 #12517
• [CHANGE] Query-frontend: Remove experimental instant query splitting feature. #12267
• [CHANGE] Query-frontend, querier: Replace query-frontend.prune-queries flag with querier.mimir-query-engine.enable-prune-toggles as pruning middleware has been moved into MQE. #12303 #12375
• [CHANGE] Distributor: Remove deprecated global HA tracker timeout configuration flags. #12321
• [CHANGE] Query-frontend: Use the Mimir Query Engine (MQE) by default. #12361
• [CHANGE] Query-frontend: Remove the CLI flags -querier.frontend-address, -querier.max-outstanding-requests-per-tenant, and -query-frontend.querier-forget-delay and corresponding YAML configurations. This is part of a change that makes the query-scheduler a required component. This removes the ability to run the query-frontend with an embedded query-scheduler. Instead, you must run a dedicated query-scheduler component. #12200
• [CHANGE] Ingester: Remove deprecated -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option. #12615
• [CHANGE] Querier: Require non-zero values for -querier.streaming-chunks-per-ingester-buffer-size and -querier.streaming-chunks-per-store-gateway-buffer-size CLI flags and corresponding YAML configurations. This is part of a change that makes streaming required between queriers, ingesters, and store-gateways. Streaming has been the default since Mimir 2.14. #12790 #12818 #12897 #12929 #12973
• [CHANGE] Remove support for the experimental read-write deployment mode. #12584
• [CHANGE] Store-gateway: Update default value of -store-gateway.dynamic-replication.multiple to 5 to increase replication of recent blocks. #12433
• [CHANGE] Cost attribution: Reduce the default maximum per-user cardinality of cost attribution labels to 2000. #12625
• [CHANGE] Querier, query-frontend: Add _total suffix to cortex_mimir_query_engine_common_subexpression_elimination_duplication_nodes_introduced, cortex_mimir_query_engine_common_subexpression_elimination_selectors_eliminated and cortex_mimir_query_engine_common_subexpression_elimination_selectors_inspected metric names. #12636
• [CHANGE] Distributor: Remove the experimental setting service_overload_status_code_on_rate_limit_enabled which used an HTTP 529 error (non-standard) instead of HTTP 429 for rate limiting. #13012
• [CHANGE] Alertmanager: Change the severity for InitialSyncFailed from 'critical' to 'warning'. #12824
• [CHANGE] Ingester: Renamed experimental reactive limiter options. #12773
• [CHANGE] Distributor: gRPC errors with the mimirpb.ERROR_CAUSE_INSTANCE_LIMIT cause are now mapped to codes.Unavailable and http.StatusServiceUnavailable instead of codes.Internal and http.StatusInternalServerError. #13003 #13032
• [CHANGE] Admin: use relative links instead of absolute ones in the administration web UI. #13034
• [CHANGE] Distributor: Use memberlist by default for the HA tracker. #12998
• [CHANGE] Block-builder: Remove cortex_blockbuilder_process_partition_duration_seconds metric and related dashboard panels. #12631
• [FEATURE] Ingester: Expose the number of active series ingested via OTLP as cortex_ingester_active_otlp_series. #12678
• [FEATURE] Distributor, ruler: Add experimental -validation.name-validation-scheme flag to specify the validation scheme for metric and label names. #12215
• [FEATURE] Ruler: Add support to use a Prometheus-compatible HTTP endpoint for remote rule evaluation. See remote evaluation mode for more details. This feature can be used to federate data from multiple Mimir instances. #11415 #11833
• [FEATURE] Distributor: Add experimental -distributor.otel-translation-strategy flag to support configuring the metric and label name translation strategy in the OTLP endpoint. #12284 #12306 #12369
• [FEATURE] Query-frontend: Add query-frontend.rewrite-propagate-matchers flag that enables a new MQE AST optimization pass that copies relevant label matchers across binary operations. #12304
• [FEATURE] Query-frontend: Add query-frontend.rewrite-histogram-queries flag that enables a new MQE AST optimization pass that rewrites histogram queries for a more efficient order of execution. #12305
• [FEATURE] Query-frontend: Support delayed name removal (Prometheus experimental feature) in MQE. #12509
• [FEATURE] Usage-tracker: Introduce a new experimental service to enforce active series limits before Kafka ingestion. #12358 #12895 #12940 #12942 #12970
• [FEATURE] Ingester: Add experimental -include-tenant-id-in-profile-labels flag to include tenant ID in pprof profiling labels for sampled traces. Currently only supported by the ingester. This can help debug performance issues for specific tenants. #12404
• [FEATURE] Alertmanager: Add experimental -alertmanager.storage.state-read-timeout flag to configure the timeout for reading the Alertmanager state (notifica...

This release contains 3 PRs from 3 authors. Thank you!

Changelog

2.16.2

Grafana Mimir

• [BUGFIX] Update to Go v1.23.12 to address CVE-2025-22871, CVE-2025-4673, CVE-2025-0913. #12582
• [BUGFIX] Update Docker base images for tools from alpine:3.21.3 to alpine:3.21.5 to address CVE-2025-9230, CVE-2025-9231, CVE-2025-2025-9232. #12990

All changes in this release: mimir-2.16.1...mimir-2.16.2

This release contains 578 PRs from 62 authors, including new contributors Andrew Hall, Frank Kloeker, Gregor Zeitlinger, Ilia Lazebnik, Julien, Maruth Goyal, Matt Clegg, Mattia Barbon, Mohijeet, Oleg Vorobev, Oliver Herrmann, Phil Kates, Pranshu Raj, Rob B, Timon Engelke, Vadim Stepanov, dannyc-grafana, unkls ben. Thank you!

Grafana Mimir version 3.0.0-rc.0 release notes

Grafana Labs is excited to announce version 3.0 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Grafana Mimir version 3.0 includes the following key features and enhancements.

Ingest storage architecture

Mimir 3.0 introduces ingest storage architecture, a next-generation architecture that separates the read and write paths using a Kafka-based ingest storage layer. In previous versions, the ingester handled both ingestion and querying, which could cause contention under heavy query loads. By introducing Kafka as an asynchronous buffer between ingestion and query, reads and writes now scale independently. This design improves ingestion stability, query performance, and operational flexibility for large deployments.

Note that in Grafana Mimir version 3.0, classic architecture is still supported.

Mimir Query Engine (MQE)

The Mimir Query Engine (MQE) is now the default query engine for all queriers and query-frontends. MQE is fully PromQL-compatible and processes data in a streaming fashion, reducing peak memory usage by up to 92% compared to the Prometheus engine. You can expect faster, more stable query performance and lower resource consumption under load.

Additional improvements

Grafana Mimir 3.0 also includes:

• Optimizations to MQE for histogram queries and query rewrites
• Improvements to query planning in frontends and queriers
• General stability and performance updates across core components

Important changes

Grafana Mimir 3.0 introduces several updates that change default behavior and configuration. Review these changes before upgrading:

• Ingest storage architecture: Grafana Mimir now uses a Kafka-based ingest storage layer that separates reads and writes. This change improves performance by altering data flow between components.
• MQE as the default engine: MQE replaces the Prometheus engine as the default query engine. To continue using the Prometheus engine, set -querier.query-engine=prometheus.
• Removal of experimental features: Some deprecated and experimental flags, including the read-write deployment mode, have been removed.
• Configuration updates: Some deprecated CLI flags and configuration options have been removed or renamed.

Experimental features

Grafana Mimir 3.0 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol
• PromQL duration expressions that allow simple arithmetic in range or offset values
• Cluster validation for HTTP requests (-server.cluster-validation.* flags)
• Native OTLP delta metric ingestion (-distributor.otel-native-delta-ingestion)

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir Helm chart is released independently. Refer to the Grafana Mimir Helm chart documentation.

Changelog

3.0.0-rc.0

Grafana Mimir

• [CHANGE] Build: Include updated Mozilla CA bundle from Debian Testing. #12247
• [CHANGE] Query-frontend: Add support for UTF-8 label and metric names in /api/v1/cardinality/{label_values|label_values|active_series} endpoints. #11848.
• [CHANGE] Querier: Add support for UTF-8 label and metric names in label_join, label_replace and count_values PromQL functions. #11848.
• [CHANGE] Remove support for Redis as a cache backend. #12163
• [CHANGE] Memcached: Remove experimental -<prefix>.memcached.addresses-provider flag to use alternate DNS service discovery backends. The more reliable backend introduced in 2.16.0 (#10895) is now the default. As a result of this change, DNS-based cache service discovery no longer supports search domains. #12175 #12385
• [CHANGE] Query-frontend: Remove the CLI flag -query-frontend.downstream-url and corresponding YAML configuration and the ability to use the query-frontend to proxy arbitrary Prometheus backends. #12191 #12517
• [CHANGE] Query-frontend: Remove experimental instant query splitting feature. #12267
• [CHANGE] Query-frontend, querier: Replace query-frontend.prune-queries flag with querier.mimir-query-engine.enable-prune-toggles as pruning middleware has been moved into MQE. #12303 #12375
• [CHANGE] Distributor: Remove deprecated global HA tracker timeout configuration flags. #12321
• [CHANGE] Query-frontend: Use the Mimir Query Engine (MQE) by default. #12361
• [CHANGE] Query-frontend: Remove the CLI flags -querier.frontend-address, -querier.max-outstanding-requests-per-tenant, and -query-frontend.querier-forget-delay and corresponding YAML configurations. This is part of a change that makes the query-scheduler a required component. This removes the ability to run the query-frontend with an embedded query-scheduler. Instead, you must run a dedicated query-scheduler component. #12200
• [CHANGE] Ingester: Remove deprecated -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option. #12615
• [CHANGE] Querier: Require non-zero values for -querier.streaming-chunks-per-ingester-buffer-size and -querier.streaming-chunks-per-store-gateway-buffer-size CLI flags and corresponding YAML configurations. This is part of a change that makes streaming required between queriers, ingesters, and store-gateways. Streaming has been the default since Mimir 2.14. #12790 #12818 #12897 #12929 #12973
• [CHANGE] Remove support for the experimental read-write deployment mode. #12584
• [CHANGE] Store-gateway: Update default value of -store-gateway.dynamic-replication.multiple to 5 to increase replication of recent blocks. #12433
• [CHANGE] Cost attribution: Reduce the default maximum per-user cardinality of cost attribution labels to 2000. #12625
• [CHANGE] Querier, query-frontend: Add _total suffix to cortex_mimir_query_engine_common_subexpression_elimination_duplication_nodes_introduced, cortex_mimir_query_engine_common_subexpression_elimination_selectors_eliminated and cortex_mimir_query_engine_common_subexpression_elimination_selectors_inspected metric names. #12636
• [CHANGE] Distributor: Remove the experimental setting service_overload_status_code_on_rate_limit_enabled which used an HTTP 529 error (non-standard) instead of HTTP 429 for rate limiting. #13012
• [CHANGE] Alertmanager: Change the severity for InitialSyncFailed from 'critical' to 'warning'. #12824
• [CHANGE] Ingester: Renamed experimental reactive limiter options. #12773
• [CHANGE] Distributor: gRPC errors with the mimirpb.ERROR_CAUSE_INSTANCE_LIMIT cause are now mapped to codes.Unavailable and http.StatusServiceUnavailable instead of codes.Internal and http.StatusInternalServerError. #13003 #13032
• [CHANGE] Admin: use relative links instead of absolute ones in the administration web UI. #13034
• [CHANGE] Distributor: Use memberlist by default for the HA tracker. #12998
• [CHANGE] Block-builder: Remove cortex_blockbuilder_process_partition_duration_seconds metric and related dashboard panels. #12631
• [FEATURE] Ingester: Expose the number of active series ingested via OTLP as cortex_ingester_active_otlp_series. #12678
• [FEATURE] Distributor, ruler: Add experimental -validation.name-validation-scheme flag to specify the validation scheme for metric and label names. #12215
• [FEATURE] Ruler: Add support to use a Prometheus-compatible HTTP endpoint for remote rule evaluation. See remote evaluation mode for more details. This feature can be used to federate data from multiple Mimir instances. #11415 #11833
• [FEATURE] Distributor: Add experimental -distributor.otel-translation-strategy flag to support configuring the metric and label name translation strategy in the OTLP endpoint. #12284 #12306 #12369
• [FEATURE] Query-frontend: Add query-frontend.rewrite-propagate-matchers flag that enables a new MQE AST optimization pass that copies relevant label matchers across binary operations. #12304
• [FEATURE] Query-frontend: Add query-frontend.rewrite-histogram-queries flag that enables a new MQE AST optimization pass that rewrites histogram queries for a more efficient order of execution. #12305
• [FEATURE] Query-frontend: Support delayed name removal (Prometheus experimental feature) in MQE. #12509
• [FEATURE] Usage-tracker: Introduce a new experimental service to enforce active series limits before Kafka ingestion. #12358 #12895 #12940 #12942 #12970
• [FEATURE] Ingester: Add experimental -include-tenant-id-in-profile-labels flag to include tenant ID in pprof profiling labels for sampled traces. Currently only supported by the ingester. This can help debug performance issues for specific tenants. #12404
• [FEATURE] Alertmanager: Add experimental -alertmanager.storage.state-read-timeout flag to configure the timeout for reading the Alertmanager state (notification log, silences) from object storage during the initial sync. #12425
• [FEATURE] Ingester: Add experimental -blocks-storage.tsdb.index-lookup-planning.* flags to configure use of a cost-based index lookup planner. This should reduce the cost...

This release contains 7 PRs from 5 authors. Thank you!

Changelog

2.17.1

Grafana Mimir

• [BUGFIX] Ingester: Fix a bug ingesters would get stuck in read-only mode after compactions. #12538
• [BUGFIX] Update to Go v1.24.6 to address CVE-2025-4674, CVE-2025-47907. #12580

All changes in this release: mimir-2.17.0...mimir-2.17.1

This release contains 735 PRs from 78 authors, including new contributors Bernaud Vincent, Carrie Edwards, danieleandreatta, David Vávra, Edgaras Giedrė, Gabija Bruzgaitė, Henrique Lourenço, Innokentii Konstantinov, Jasper Maes, Jeanette Tan, Joobi S B, Julius Hinze, Lukas Bischofberger, mihaelmiklec, mimir-vendoring[bot], Nasiel, pierremahot, rektabhi, sam clulow, Shay Pletcher, Thor K. Høgås, Toni Cárdenas, Yuran Ou, zhuoyuan-liu. Thank you!

Grafana Mimir version 2.17.0 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0

Grafana Mimir

• [CHANGE] Query-frontend: Ensure that cache keys generated from cardinality estimate middleware are less than 250 bytes in length by hashing the tenant IDs that are included in them. This change invalidates all cardinality estimates in the cache. #11568
• [CHANGE] Ruler: Remove experimental CLI flag -ruler-storage.cache.rule-group-enabled to enable or disable caching the contents of rule groups. Caching rule group contents is now always enabled when a cache is configured for the ruler. #10949
• [CHANGE] Ingester: Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled. The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. #10956
• [CHANGE] Distributor: removed the cortex_distributor_label_values_with_newlines_total metric. #10977
• [CHANGE] Ingester/Distributor: renamed the experimental max_cost_attribution_cardinality_per_user config to max_cost_attribution_cardinality. #11092
• [CHANGE] Frontend: The subquery spin-off feature is now enabled with -query-frontend.subquery-spin-off-enabled=true instead of -query-frontend.instant-queries-with-subquery-spin-off=.* #11153
• [CHANGE] Overrides-exporter: Don't export per-tenant overrides that are set to their default values. #11173
• [CHANGE] gRPC/HTTP clients: Rename metric cortex_client_request_invalid_cluster_validation_labels_total to cortex_client_invalid_cluster_validation_label_requests_total. #11237
• [CHANGE] Querier: Use Mimir Query Engine (MQE) by default. Set -querier.query-engine=prometheus to continue using Prometheus' engine. #11501
• [CHANGE] Memcached: Ignore initial DNS resolution failure, meaning don't depend on Memcached on startup. #11602
• [CHANGE] Ingester: The -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release. #11711
• [CHANGE] Distributor: track cortex_ingest_storage_writer_latency_seconds metric for failed writes too. Added outcome label to distinguish between success and failure. #11770
• [CHANGE] Distributor: renamed few metrics used by experimental ingest storage. #11766
  • Renamed cortex_ingest_storage_writer_produce_requests_total to cortex_ingest_storage_writer_produce_records_enqueued_total
  • Renamed cortex_ingest_storage_writer_produce_failures_total to cortex_ingest_storage_writer_produce_records_failed_total
• [CHANGE] Distributor: moved HA tracker timeout config to limits. #11774
  • Moved distributor.ha_tracker.ha_tracker_update_timeout to limits.ha_tracker_update_timeout.
  • Moved distributor.ha_tracker.ha_tracker_update_timeout_jitter_max to limits.ha_tracker_update_timeout_jitter_max.
  • Moved distributor.ha_tracker.ha_tracker_failover_timeout to limits.ha_tracker_failover_timeout.
• [CHANGE] Distributor: Memberlist marked as stable as an option for backend storage for the HA tracker. #11861
• [CHANGE] Distributor: etcd deprecated as an option for backend storage for the HA tracker. #12047
• [CHANGE] Memberlist: Apply new default configuration values for MemberlistKV. This unlocks using it as backend storage for the HA Tracker. We have observed better performance with these defaults across different production loads. #11874
  • memberlist.packet-dial-timeout: 500ms
  • memberlist.packet-write-timeout: 500ms
  • memberlist.max-concurrent-writes: 5
  • memberlist.acquire-writer-timeout: 1s
    These defaults perform better but may cause long-running packets to be dropped in high-latency networks.
• [CHANGE] Query-frontend: Apply query pruning and check for disabled experimental functions earlier in query processing. #11939
• [FEATURE] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol. Limitations: Created timestamp is ignored, per series metadata is merged on metric family level automatically, ingestion might fail if client sends ProtoBuf fields out of order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0 depending on the detected Remote-Write protocol. #11100 #11101 #11192 #11143
• [FEATURE] Query-frontend: expand query-frontend.cache-errors and query-frontend.results-cache-ttl-for-errors configuration options to cache non-transient response failures for instant queries. #11120
• [FEATURE] Query-frontend: Allow use of Mimir Query Engine (MQE) via the experimental CLI flags -query-frontend.query-engine or -query-frontend.enable-query-engine-fallback or corresponding YAML. #11417 #11775
• [FEATURE] Querier, query-frontend, ruler: Enable experimental support for duration expressions in Pro...

This release contains 7 PRs from 4 authors. Thank you!

Grafana Mimir version 2.17.0-rc.2 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.2

Grafana Mimir

• [ENHANCEMENT] Stagger head compaction intervals across zones to prevent compactions from aligning simultaneously, which could otherwise cause strong consistency queries to fail when experimental ingest storage is enabled. #12090
• [ENHANCEMENT] Compactor: Add -compactor.update-blocks-concurrency flag to control concurrency for updating block metadata during bucket index updates, separate from deletion marker concurrency. #12117
• [ENHANCEMENT] Query-frontend: Allow users to set the query-frontend.extra-propagated-headers flag to specify the extra headers allowed to pass through to the rest of the query path. #12174
• [BUGFIX] Ingester: Fix issue where ingesters can exit read-only mode during idle compactions, resulting in write errors. #12128
• [BUGFIX] otlp: Reverts #11889 which has a pooled memory re-use bug. #12266

All changes in this release: mimir-2.17.0-rc.1...mimir-2.17.0-rc.2

This release contains 81 PRs from 27 authors, including new contributors Edgaras Giedrė, Julius Hinze. Thank you!

Grafana Mimir version 2.17.0-rc.1 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.1

Grafana Mimir

• [FEATURE] Distributor: Add experimental -distributor.otel-native-delta-ingestion option to allow primitive delta metrics ingestion via the OTLP endpoint. #11631
• [FEATURE] MQE: Add support for experimental sort_by_label and sort_by_label_desc PromQL functions. #11930
• [FEATURE] Ingester/Block-builder: Handle the created timestamp field for remote-write requests. #11977
• [FEATURE] Cost attribution: Labels specified in the limit configuration may specify an output label in order to override emitted label names. #12035
• [ENHANCEMENT] Ingester: Display user grace interval in the tenant list obtained through the /ingester/tenants endpoint. #11961
• [ENHANCEMENT] kafkatool: add consumer-group delete-offset command as a way to delete the committed offset for a consumer group. #11988
• [ENHANCEMENT] Block-builder-scheduler: Detect gaps in scheduled and completed jobs. #11867
• [ENHANCEMENT] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol has been updated. Created timestamps are now supported. This feature includes some limitations. If samples in a write request aren't ordered by time, the created timestamp might be dropped. Additionally, per-series metadata is automatically merged on the metric family level. Ingestion might fail if the client sends ProtoBuf fields out-of-order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0, depending on the detected remote-write protocol. #11977
• [ENHANCEMENT] Query-frontend: Added labels query optimizer that automatically removes redundant __name__!="" matchers from label names and label values queries, improving query performance. You can enable the optimizer per-tenant with the labels_query_optimizer_enabled runtime configuration flag. #12054 #12066 #12076 #12080
• [ENHANCEMENT] Query-frontend: Standardise non-regex patterns in query blocking upon loading of config. #12102
• [ENHANCEMENT] Ruler: Propagate GCS object mutation rate limit for rule group uploads. #12086
• [BUGFIX] Distributor: Validate the RW2 symbols field and reject invalid requests that don't have an empty string as the first symbol. #11953
• [BUGFIX] Distributor: Check max_inflight_push_requests_bytes before decompressing incoming requests. #11967
• [BUGFIX] Query-frontend: Allow limit parameter to be 0 in label queries to explicitly request unlimited results. #12054
• [BUGFIX] Distributor: Fix a possible panic in the OTLP push path while handling a gRPC status error. #12072
• [BUGFIX] Query-frontend: Evaluate experimental duration expressions before sharding, splitting, and caching. Otherwise, the result is not correct. #12038
• [BUGFIX] Block-builder-scheduler: Fix bugs in handling of partitions with no commit. #12130

Mixin

• [CHANGE] Remove support for the experimental read-write deployment mode. #11975
• [CHANGE] Alerts: Replace namespace with job label in golang_alerts. #11957
• [FEATURE] Add an alert if the block-builder-scheduler detects that it has skipped data. #12118

Jsonnet

• [CHANGE] Removed support for the experimental read-write deployment mode. #11974
• [ENHANCEMENT] Add assertion to ensure ingester ScaledObject has minimum and maximum replicas set to a value greater than 0. #11979
• [ENHANCEMENT] Add ingest_storage_migration_ignore_ingest_storage_errors and ingest_storage_migration_ingest_storage_max_wait_time configs to control error handling of the partition ingesters during ingest storage migrations. #12105
• [ENHANCEMENT] Add block-builder job processing duration timings and offset-skipped errors to the Block-builder dashboard. #12118

Documentation

• [ENHANCEMENT] Update the MimirIngestedDataTooFarInTheFuture runbook with a note about false positives and the endpoint to flush TSDB blocks by user. #11961

All changes in this release: mimir-2.17.0-rc.0...mimir-2.17.0-rc.1