This is release v1.11.3 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Enhancements

• Schedule new path targets faster in loki.source.file. (@kalleep)

• Add prometheus.static.exporter that exposes metrics specified in a text file in Prometheus exposition format. (@kalleep)

Bugfixes

• local.file_match now publish targets faster whenever targets in arguments changes. (@kalleep)

• Fix otelcol.exporter.splunkhec arguments missing documented otel_attrs_to_hec_metadata block. (@dehaansa)

• Support Scrape Protocol specification in CRDS for prometheus.operator.* components. (@dehaansa)

• Fix panic in otelcol.receiver.syslog when no tcp block was configured. (@kalleep)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.11.2 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Bugfixes

• Fix potential deadlock in loki.source.journal when stopping or reloading the component. (@thampiotr)

• Honor sync timeout when waiting for network availability for prometheus.operator.* components. (@dehaansa)

• Fix prometheus.exporter.cloudwatch to not always emit debug logs but respect debug property. (@kalleep)

• Fix an issue where component shutdown could block indefinitely by adding a warning log message and a deadline of 10 minutes. The deadline can be configured with the --feature.component-shutdown-deadline flag if the default is not suitable. (@thampiotr)

• Fix potential deadlocks in loki.source.file and loki.source.journal when component is shutting down. (@kalleep, @thampiotr)

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.11.0 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Breaking changes

• Prometheus dependency had a major version upgrade from v2.55.1 to v3.4.2. (@thampiotr)

  • The . pattern in regular expressions in PromQL matches newline characters now. With this change a regular expressions like .* matches strings that include \n. This applies to matchers in queries and relabel configs in Prometheus and Loki components.

  • The enable_http2 in prometheus.remote_write component's endpoints has been changed to false by default. Previously, in Prometheus v2 the remote write http client would default to use http2. In order to parallelize multiple remote write queues across multiple sockets its preferable to not default to http2. If you prefer to use http2 for remote write you must now set enable_http2 to true in your prometheus.remote_write endpoints configuration section.

  • The experimental CLI flag --feature.prometheus.metric-validation-scheme has been deprecated and has no effect. You can configure the metric validation scheme individually for each prometheus.scrape component.

  • Log message format has changed for some of the prometheus.* components as part of the upgrade to Prometheus v3.

  • The values of the le label of classic histograms and the quantile label of summaries are now normalized upon ingestion. In previous Alloy versions, that used Prometheus v2, the value of these labels depended on the scrape protocol (protobuf vs text format) in some situations. This led to label values changing based on the scrape protocol. E.g. a metric exposed as my_classic_hist{le="1"} would be ingested as my_classic_hist{le="1"} via the text format, but as my_classic_hist{le="1.0"} via protobuf. This changed the identity of the metric and caused problems when querying the metric. In current Alloy release, which uses Prometheus v3, these label values will always be normalized to a float like representation. I.e. the above example will always result in my_classic_hist{le="1.0"} being ingested into Prometheus, no matter via which protocol. The effect of this change is that alerts, recording rules and dashboards that directly reference label values as whole numbers such as le="1" will stop working.

    The recommended way to deal with this change is to fix references to integer le and quantile label values, but otherwise do nothing and accept that some queries that span the transition time will produce inaccurate or unexpected results.

  See the upstream Prometheus v3 migration guide for more details.

• prometheus.exporter.windows dependency has been updated to v0.31.1. (@dehaansa)

  • There are various renamed metrics and two removed collectors (cs, logon).

• scrape_native_histograms attribute for prometheus.scrape is now set to false, whereas in previous versions of Alloy it would default to true. This means that it is no longer enough to just configure scrape_protocols to start with PrometheusProto to scrape native histograms - scrape_native_histograms has to be enabled. If scrape_native_histograms is enabled, scrape_protocols will automatically be configured correctly for you to include PrometheusProto. If you configure it explicitly, Alloy will validate that PrometheusProto is in the scrape_protocols list.

• Add otel_attrs_to_hec_metadata configuration block to otelcol.exporter.splunkhec to match otelcol.receiver.splunkhec. (@cgetzen)

• [otelcol.processor.batch] Two arguments have different default values. (@ptodev)

  • send_batch_size is now set to 2000 by default. It used to be 8192.
  • send_batch_max_size is now set to 3000 by default. It used to be 0.
  • This helps prevent issues with ingestion of batches that are too large.

• OpenTelemetry Collector dependencies upgraded from v0.128.0 to v0.134.0. (@ptodev)

  • The otelcol.receiver.opencensus component has been deprecated and will be removed in a future release, use otelcol.receiver.otelp instead.
  • [otelcol.exporter.*] The deprecated blocking argument in the sending_queue block has been removed.
    Use block_on_overflow instead.
  • [otelcol.receiver.kafka, otelcol.exporter.kafka]: Removed the broker_addr argument from the aws_msk block.
    Also removed the SASL/AWS_MSK_IAM authentication mechanism.
  • [otelcol.exporter.splunkhec] The batcher block is deprecated and will be removed in a future release. Use the queue block instead.
  • [otelcol.exporter.loadbalancing] Use a linear probe to decrease variance caused by hash collisions, which was causing a non-uniform distribution of loadbalancing.
  • [otelcol.connector.servicegraph] The database_name_attribute argument has been removed.
  • [otelcol.connector.spanmetrics] Adds a default maximum number of exemplars within the metric export interval.
  • [otelcol.processor.tail_sampling] Add a new block_on_overflow config attribute.

Features

• Add the otelcol.receiver.fluentforward receiver to receive logs via Fluent Forward Protocol. (@rucciva)

• Add the prometheus.enrich component to enrich metrics using labels from discovery.* components. (@ArkovKonstantin)

• Add node_filter configuration block to loki.source.podlogs component to enable node-based filtering for pod discovery. When enabled, only pods running on the specified node will be discovered and monitored, significantly reducing API server load and network traffic in DaemonSet deployments. (@QuentinBisson)

• (Experimental) Additions to experimental database_observability.mysql component:

  • query_sample collector now supports auto-enabling the necessary setup_consumers settings (@cristiangreco)
  • query_sample collector is now compatible with mysql less than 8.0.28 (@cristiangreco)
  • include server_id label on log entries (@matthewnolf)
  • support receiving targets argument and relabel those to include server_id (@matthewnolf)
  • updated the config blocks and documentation (@cristiangreco)

• (Experimental) Additions to experimental database_observability.postgres component:

  • add query_tables collector for postgres (@matthewnolf)
  • add cloud_provider.aws configuration that enables optionally supplying the ARN of the database under observation. The ARN is appended to metric samples as labels for easier filtering and grouping of resources.
  • add query_sample collector for postgres (@gaantunes)
  • add schema_table collector for postgres (@fridgepoet)
  • include server_id label on logs and metrics (@matthewnolf)

• Add otelcol.receiver.googlecloudpubsub community component to receive metrics, traces, and logs from Google Cloud Pub/Sub subscription. (@eraac)

• (Experimental) Add a honor_metadata configuration argument to the prometheus.scrape component.
  When set to true, it will propagate metric metadata to downstream components.

• Add a flag to pyroscope.ebpf alloy configuration to set the off-cpu profiling threshold. (@luweglarz)

• Add encoding.url_encode and encoding.url_decode std lib functions. (@kalleep)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.11.0-rc.3 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Bugfixes

• Fix graph UI so it generates correct URLs for components in remotecfg modules. (@patrickeasters)

• Fix panic in loki.write when component is shutting down and external_labels are configured. (@kalleep)

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.11.0-rc.2 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Bugfixes

• Use new cache for metadata cache in prometheus.write.queue and support disabling the metadata cache with it disable by default. (@kgeckhart, @dehaansa)

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.11.0-rc.1 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Bugfixes

• Fix loki.source.podlogs component to properly collect logs from Kubernetes Jobs and CronJobs. Previously, the component would fail to scrape logs from short-lived or terminated jobs due to race conditions between job completion and pod discovery. The fix includes:

  • Job-aware termination logic with extended grace periods (10-60 seconds) to ensure all logs are captured
  • Proper handling of pod deletion and race conditions between job completion and controller cleanup
  • Separation of concerns: shouldStopTailingContainer() handles standard Kubernetes restart policies for regular pods, while shouldStopTailingJobContainer() handles job-specific lifecycle with grace periods
  • Enhanced deduplication mechanisms to prevent duplicate log collection while ensuring comprehensive coverage
  • Comprehensive test coverage including unit tests and deduplication validation
    This resolves the issue where job logs were being missed, particularly for fast-completing jobs or jobs that terminated before discovery. (@QuentinBisson)

• Fix prometheus.exporter.redis component so that it no longer ignores the MaxDistinctKeyGroups configuration option. If key group metrics are enabled, this will increase the cardinality of the generated metrics. (@stegosaurus21)

• Fix loki.source.journal creation failing with an error when the journal file is not found. (@thampiotr)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.11.0-rc.0 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Breaking changes

• Prometheus dependency had a major version upgrade from v2.55.1 to v3.4.2. (@thampiotr)

  • The . pattern in regular expressions in PromQL matches newline characters now. With this change a regular expressions like .* matches strings that include \n. This applies to matchers in queries and relabel configs in Prometheus and Loki components.

  • The enable_http2 in prometheus.remote_write component's endpoints has been changed to false by default. Previously, in Prometheus v2 the remote write http client would default to use http2. In order to parallelize multiple remote write queues across multiple sockets its preferable to not default to http2. If you prefer to use http2 for remote write you must now set enable_http2 to true in your prometheus.remote_write endpoints configuration section.

  • The experimental CLI flag --feature.prometheus.metric-validation-scheme has been deprecated and has no effect. You can configure the metric validation scheme individually for each prometheus.scrape component.

  • Log message format has changed for some of the prometheus.* components as part of the upgrade to Prometheus v3.

  • The values of the le label of classic histograms and the quantile label of summaries are now normalized upon ingestion. In previous Alloy versions, that used Prometheus v2, the value of these labels depended on the scrape protocol (protobuf vs text format) in some situations. This led to label values changing based on the scrape protocol. E.g. a metric exposed as my_classic_hist{le="1"} would be ingested as my_classic_hist{le="1"} via the text format, but as my_classic_hist{le="1.0"} via protobuf. This changed the identity of the metric and caused problems when querying the metric. In current Alloy release, which uses Prometheus v3, these label values will always be normalized to a float like representation. I.e. the above example will always result in my_classic_hist{le="1.0"} being ingested into Prometheus, no matter via which protocol. The effect of this change is that alerts, recording rules and dashboards that directly reference label values as whole numbers such as le="1" will stop working.

    The recommended way to deal with this change is to fix references to integer le and quantile label values, but otherwise do nothing and accept that some queries that span the transition time will produce inaccurate or unexpected results.

  See the upstream Prometheus v3 migration guide for more details.

• prometheus.exporter.windows dependency has been updated to v0.31.1. (@dehaansa)

  • There are various renamed metrics and two removed collectors (cs, logon).

• Add otel_attrs_to_hec_metadata configuration block to otelcol.exporter.splunkhec to match otelcol.receiver.splunkhec. (@cgetzen)

• [otelcol.processor.batch] Two arguments have different default values. (@ptodev)

  • send_batch_size is now set to 2000 by default. It used to be 8192.
  • send_batch_max_size is now set to 3000 by default. It used to be 0.
  • This helps prevent issues with ingestion of batches that are too large.

• OpenTelemetry Collector dependencies upgraded from v0.128.0 to v0.134.0. (@ptodev)

  • The otelcol.receiver.opencensus component has been deprecated and will be removed in a future release, use otelcol.receiver.otelp instead.
  • [otelcol.exporter.*] The deprecated blocking argument in the sending_queue block has been removed.
    Use block_on_overflow instead.
  • [otelcol.receiver.kafka, otelcol.exporter.kafka]: Removed the broker_addr argument from the aws_msk block.
    Also removed the SASL/AWS_MSK_IAM authentication mechanism.
  • [otelcol.exporter.splunkhec] The batcher block is deprecated and will be removed in a future release. Use the queue block instead.
  • [otelcol.exporter.loadbalancing] Use a linear probe to decrease variance caused by hash collisions, which was causing a non-uniform distribution of loadbalancing.
  • [otelcol.connector.servicegraph] The database_name_attribute argument has been removed.
  • [otelcol.connector.spanmetrics] Adds a default maximum number of exemplars within the metric export interval.
  • [otelcol.processor.tail_sampling] Add a new block_on_overflow config attribute.

Features

• Add the otelcol.receiver.fluentforward receiver to receive logs via Fluent Forward Protocol. (@rucciva)

• Add the prometheus.enrich component to enrich metrics using labels from discovery.* components. (@ArkovKonstantin)

• Add node_filter configuration block to loki.source.podlogs component to enable node-based filtering for pod discovery. When enabled, only pods running on the specified node will be discovered and monitored, significantly reducing API server load and network traffic in DaemonSet deployments. (@QuentinBisson)

• (Experimental) Additions to experimental database_observability.mysql component:

  • query_sample collector now supports auto-enabling the necessary setup_consumers settings (@cristiangreco)
  • query_sample collector is now compatible with mysql less than 8.0.28 (@cristiangreco)
  • include server_id label on log entries (@matthewnolf)
  • support receiving targets argument and relabel those to include server_id (@matthewnolf)
  • updated the config blocks and documentation (@cristiangreco)

• (Experimental) Additions to experimental database_observability.postgres component:

  • add query_tables collector for postgres (@matthewnolf)
  • add cloud_provider.aws configuration that enables optionally supplying the ARN of the database under observation. The ARN is appended to metric samples as labels for easier filtering and grouping of resources.
  • add query_sample collector for postgres (@gaantunes)
  • add schema_table collector for postgres (@fridgepoet)
  • include server_id label on logs and metrics (@matthewnolf)

• Add otelcol.receiver.googlecloudpubsub community component to receive metrics, traces, and logs from Google Cloud Pub/Sub subscription. (@eraac)

• (Experimental) Add a honor_metadata configuration argument to the prometheus.scrape component.
  When set to true, it will propagate metric metadata to downstream components.

• Add a flag to pyroscope.ebpf alloy configuration to set the off-cpu profiling threshold. (@luweglarz)

• Add encoding.url_encode and encoding.url_decode std lib functions. (@kalleep)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.10.2 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Bugfixes

• Fix issue in prometheus.write.queue causing inability to increase shard count if existing WAL data was present on start. (@kgeckhart)

• Fix issue with loki.source.gcplog when push messages sent by gcp pub/sub only includes messageId. (@kalleep)

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.10.1 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions:

Notable changes:

Bugfixes

• Fix issue with faro.receiver cors not allowing X-Scope-OrgID and traceparent headers. (@mar4uk)

• Fix issues with propagating cluster peers change notifications to components configured with remotecfg. (@dehaansa)

• Fix issues with statistics reporter not including components only configured with remotecfg. (@dehaansa)

• Fix issues with prometheus.exporter.windows not propagating dns collector config. (@dehaansa)

• Fixed a bug in prometheus.write.queue which caused retries even when max_retry_attempts was set to 0. (@ptodev)

• Fixed a bug in prometheus.write.queue which caused labelling issues when providing more than one label in external_labels. (@dehaansa)

• Add application_host and network_inter_zone features to beyla.ebpf component. (@marctc)

• Fix issues in loki.process where stage.multiline did not pass through structured metadata. (@jan-mrm)

• Fix URLs in the Windows installer being wrapped in quotes. (@martincostello)

• Fixed an issue where certain otelcol.* components could prevent Alloy from shutting down when provided invalid configuration. (@thampiotr)

For a full list of changes, please refer to the CHANGELOG

Installation

Refer to our installation guide for how to install Grafana Alloy.

This is release v1.10.0 of Grafana Alloy.

Upgrading

Read the release notes for specific instructions on upgrading from older versions.

Notable changes

Breaking changes

• Removing the nanoserver-1809 container image for Windows 2019. (@ptodev)
  This is due to the deprecation of windows-2019 GitHub Actions runners.
  The windowsservercore-ltsc2022 Alloy image is still being published to DockerHub.

Bugfixes

• Upgrade otelcol components from OpenTelemetry v0.126.0 to v0.128.0 (@korniltsev, @dehaansa)
  • [otelcol.exporter.kafka]: Allow kafka exporter to produce to topics based on metadata key values.
  • [otelcol.receiver.kafka]: Enforce a backoff mechanism on non-permanent errors, such as when the queue is full.
  • [otelcol.receiver.kafka]: Don't restart the Kafka consumer on failed errors when message marking is enabled for them.
  • [otelcol.connector.spanmetrics]: Fix bug causing span metrics calls count to be always 0 when using delta temporality.

Features

• (Experimental) Add an array.group_by stdlib function to group items in an array by a key. (@wildum)

• Add the otelcol.exporter.faro exporter to export traces and logs to Faro endpoint. (@mar4uk)

• Add the otelcol.receiver.faro receiver to receive traces and logs from the Grafana Faro Web SDK. (@mar4uk)

• Add entropy support for loki.secretfilter (@romain-gaillard)

Enhancements

• Add hash_string_id argument to foreach block to hash the string representation of the pipeline id instead of using the string itself. (@wildum)

• (Experimental) Additions to experimental database_observability.mysql component:

  • Add explain_plan collector to database_observability.mysql component. (@rgeyer)
  • locks: addition of data locks collector (@gaantunes @fridgepoet)
  • Query sample collector is now enabled by default (@matthewnolf)

• (Experimental) prometheus.write.queue add support for exemplars. (@dehaansa)

• Wire in survey block for beyla.ebpf component. (@grcevski, @tpaschalis)

• Upgrade otelcol components from OpenTelemetry v0.126.0 to v0.128.0 (@korniltsev, @dehaansa)

  • [otelcol.processor.resourcedetection]: Add additional OS properties to resource detection: os.build.id and os.name.
  • [otelcol.processor.resourcedetection]: Add host.interface resource attribute to system detector.
  • [otelcol.exporter.kafka]: Fix Snappy compression codec support for the Kafka exporter.
  • [otelcol.receiver.filelog]: Introduce utf8-raw encoding to avoid replacing invalid bytes with \uFFFD when reading UTF-8 input.
  • [otelcol.processor.k8sattributes]: Support extracting labels and annotations from k8s Deployments.
  • [otelcol.processor.k8sattributes]: Add option to configure automatic service resource attributes.
  • [otelcol.exporter.datadog]: Adds hostname_detection_timeout configuration option for Datadog Exporter and sets default to 25 seconds.
  • [otelcol.receiver.datadog]: Address semantic conventions noncompliance and add support for http/db.
  • [otelcol.exporter.awss3]: Add the retry mode, max attempts and max backoff to the settings.

• Add support for conditions and statement-specific error_mode in otelcol.processor.transform. (@ptodev)

• Add storage and start_from args to cloudwatch logs receiver. (@boernd)

Bugfixes

• Fix extension registration for otelcol.receiver.splunkhec auth extensions. (@dehaansa)

Other changes

• Mark pyroscope.receive_http and pyroscope.relabel components as GA. (@marcsanmi)

• Upgrade otelcol.exporter.windows to v0.30.8 to get bugfixes and fix update collector support. (@dehaansa)

For a full list of changes, please refer to the CHANGELOG!

Installation

Refer to our installation guide for how to install Grafana Alloy.