Grafana Tempo in MicroService mode

• https://github.com/grafana/tempo

Kubernetes: ^1.25.0-0

OCI registries are preferred in Helm as they implement unified storage, distribution, and improved security.

helm install RELEASE-NAME oci://ghcr.io/grafana-community/helm-charts/tempo-distributed

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm install RELEASE-NAME grafana-community/tempo-distributed

To remove all of the Kubernetes objects associated with the Helm chart release:

helm delete RELEASE-NAME

See the changelog.

A major chart version change indicates that there is an incompatible breaking change needing manual actions.

The built-in MinIO subchart has been removed in chart v3.0.0. minio is no longer a chart dependency, and setting minio.enabled: true fails the render. Tempo must point at an externally managed S3-compatible object store via storage.trace.s3:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio.minio-namespace.svc.cluster.local:9000
      access_key: <access-key>
      secret_key: <secret-key>
      insecure: true  # remove if TLS is configured

If you previously used the built-in subchart, do not run a plain helm upgrade: it would garbage collect the MinIO Deployment, Service, Secret, and PVC and destroy your trace data. Detach those objects from the release first (annotate them with helm.sh/resource-policy: keep, the Helm equivalent of kubectl delete --cascade=orphan), then repoint storage.trace.s3 at the surviving MinIO Service. See UPGRADE.md for the full procedure.

Version 2.17.10 change the memcached Services and Statefulsets spec. These changes cannot be done by patching the resources, causing an existing installation not to be upgradable without manual interaction.

An upgrade will fail with a message like:

Error: UPGRADE FAILED: Service "tempo-memcached" is invalid: spec.clusterIPs[0]: Invalid value: ["None"]: may not change once set && StatefulSet.apps "tempo-memcached" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

There are basically two options:

Uninstall the old release and re-install the new one.

Delete the affected Services and Statefulsets, and re-install the new ones.

kubectl -n <namespace> delete service --selector 'app.kubernetes.io/instance=<instance-name>,app.kubernetes.io/component in (memcached,memcached-bloom,memcached-parquet-footer,memcached-frontend-search)'

Perform a non-cascading deletion of the Statefulsets which will keep the pods running:

kubectl -n <namespace> delete statefulset --selector 'app.kubernetes.io/instance=<instance-name>,app.kubernetes.io/component in (memcached,memcached-bloom,memcached-parquet-footer,memcached-frontend-search)' --cascade=orphan

Perform a regular Helm upgrade on the existing release. The new Statefulsets will pick up the existing pods and perform a rolling upgrade.

Tempo 3.0 replaces the ingester-based write path with a Kafka-backed architecture. This is a breaking change for microservices-mode deployments.

See UPGRADE.md for the full migration guide, including the parallel-deployment path, Kafka configuration, and the tempo-cli migrate config command.

The minimum required Kubernetes version is now 1.25. All references to deprecated APIs have been removed.

Please be aware that we've updated the Tempo version to 2.9, which includes some breaking changes We recommend reviewing the release notes before upgrading.

Please be aware that we've updated the Tempo version to 2.8, which includes some breaking changes We recommend reviewing the release notes before upgrading.

• Breaking Change * In order to be consistent with other projects and documentations, the default port has been changed from 3100 to 3200.

• Breaking Change * In order to reduce confusion, the overrides configurations have been renamed as below.

global_overrides => overrides (this is where the defaults for every tenant is set) overrides => per_tenant_overrides (this is where configurations for specific tenants can be set)

Tempo serverless was deprecated in tempo 2.7 release, Config options related to serverless are being removed from helm chart, and this might be a breaking change if you were using tempo serverless.

These config optioons are removed in tempo#4599 and will not work from next release of tempo.

Please be aware that we've updated the Tempo version to 2.7, which includes some breaking changes We recommend reviewing the release notes before upgrading.

A default affinity has been defined in this version for the compactor following the standard used in other components.

Upgrading to chart 1.21.0 will set the memberlist cluster_label config option. During rollout your cluster will temporarily be split into two memberlist clusters until all components are rolled out. This will interrupt reads and writes. This config option is set to prevent cross talk between Tempo and other memberlist clusters.

Please be aware that we've updated the minor version to Tempo 2.6, which includes breaking changes. We recommend reviewing the release notes before upgrading.

Switch to new overrides format, see https://grafana.com/docs/tempo/latest/configuration/#overrides.

EXPERIMENTAL: Zone Aware Replication has been added to the ingester statefulset. Attention, the calculation of the pods per AZ is as follows (.values.ingester.replicas + numberOfZones -1)/numberOfZones

The metrics generator component in the chart has been disabled by default, but the configuration for the processors was not empty, resulting error sin the distributor log. Here we align the default metrics generator config settings to both disable the generator and remove processors. Users who wish to keep the their processors enabled, will need to update their values.

Please be aware that we've updated the minor version to Tempo 2.2, which includes breaking changes. We recommend reviewing the release notes before upgrading.

Please be aware that we've updated the minor version to Tempo 2.1, which includes breaking changes. We recommend reviewing the release notes before upgrading.

Please note that we've incremented the major version when upgrading to Tempo 2.0. There were a large number of changes in this release (breaking and otherwise). It is encouraged to review the release notes and 1.5 -> 2.0 upgrade guide before upgrading.

Version 0.27.0:

Many changes have been introduced, including some breaking changes.

The PR includes additional details.

• BREAKING CHANGE centralize selector label handling -- users who wish to keep old values should still be able to use the nameOverride and fullNameOverride top level keys in their values.

• BREAKING CHANGE serviceMonitor has been nested under metaMonitoring -- metamonitoring can be used scrape services as well as install the operator with the following values. Note also that the port names have changed from http to http-metrics.

metaMonitoring:
  serviceMonitor:
    enabled: true
  grafanaAgent:
    enabled: true
    installOperator: true

• allow configuration to be stored in a secret. See the documentation for useExternalConfig and configStorageType in the values file for more details.

Version 0.26.0

• Moves metricsGenerator.config.storage_remote_write to metricsGenerator.config.storage.remote_write
• Moves metricsGenerator.config.service_graphs_max_items to metricsGenerator.config.processor.service_graphs.max_items

Version 0.23.0:

• Adds /var/tempo emptyDir mount for querier, queryfrontend, distributor and compactor. Previously, /var/tempo was directory inside container.

• Sets queryFrontend.query.enabled to false. tempo-query is only required for grafana version <7.5 for compatibility with jaeger-ui. Please also note that tempo-query is incompatible with securityContext readOnlyRootFilesystem set to true.

• Sets stricter default securityContext:

tempo:
  securityContext:
    capabilities:
      drop:
        - ALL
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false
  podSecurityContext:
    fsGroup: 1000

If you had ingester persistence enabled, you might need to manually change ownership of files in your PV if your CSI doesn't support fsGroup

Align Istio gRPC named port syntax. For example,

• otlp-grpc -> grpc-otlp
• distributor-otlp-grpc -> grpc-distributor-otlp
• jaeger-grpc -> grpc-jaeger
• distributor-jaeger-grpc -> grpc-distributor-jaeger

In case you need to rollback, please search the right hand side pattern and replace with left hand side pattern.

The image's attributes must be set under the image key for the Memcached service.

memcached:
  image:
    registry: docker.io
    repository: memcached
    tag: "1.5.17-alpine"
    pullPolicy: "IfNotPresent"

Trace ingestion must now be enabled with the enabled key:

traces:
  otlp:
    grpc:
      enabled: true
    http:
      enabled: true
  zipkin:
    enabled: true
  jaeger:
    thriftHttp:
      enabled: true
  opencensus:
    enabled: true

This release the component label was shortened to be more aligned with the Loki-distributed chart and the mixin dashboards.

Due to the label changes, an existing installation cannot be upgraded without manual interaction. There are basically two options:

Option 1 Uninstall the old release and re-install the new one. There will be no data loss, as the collectors/agents can cache for a short period.

Option 2 Add new selector labels to the existing pods. This option will make your pods also temporarily unavailable, option 1 is faster:

kubectl label pod -n <namespace> -l app.kubernetes.io/component=<release-name>-tempo-distributed-<component>,app.kubernetes.io/instance=<instance-name> app.kubernetes.io/component=<component> --overwrite

Perform a non-cascading deletion of the Deployments and Statefulsets which will keep the pods running:

kubectl delete <deployment/statefulset> -n <namespace> -l app.kubernetes.io/component=<release-name>-tempo-distributed-<component>,app.kubernetes.io/instance=<instance-name> --cascade=false

Perform a regular Helm upgrade on the existing release. The new Deployment/Statefulset will pick up the existing pods and perform a rolling upgrade.

By default all tracing protocols are disabled and you need to specify which protocols to enable for ingestion.

For example to enable Jaeger gRPC thrift http and zipkin protocols:

traces:
  jaeger:
    grpc: true
    thriftHttp: true
  zipkin: true

The distributor service is now called {{tempo.fullname}}-distributor. That could impact your ingestion towards this service.

The memcached default args are removed and should be provided manually. The settings for the memcached.exporter moved to memcachedExporter

The chart supports the components shown in the following table. Distributor, querier, and query-frontend are always installed. The other components are optional and must be explicitly enabled.

This chart configures Tempo in microservices mode.

Refer to the Get started with Grafana Tempo using the Helm chart documentation for more details.

NOTE: In its default configuration, the chart uses local filesystem as storage. The reason for this is that the chart can be validated and installed in a CI pipeline. However, this setup is not fully functional. The recommendation is to use object storage, such as S3, GCS, MinIO, etc., or one of the other options documented at https://grafana.com/docs/tempo/latest/configuration/#storage.

Alternatively, in order to quickly test Tempo using the filestore, the single binary chart can be used.

tempo.structuredConfig variable can be used to alter individual values in the configuration and it's structured YAML instead of text. It takes precedence over all other variable adjustments inside tempo.yaml config file, ie s3 storage settings.

Example:

tempo:
  structuredConfig:
    query_frontend:
      search:
        max_duration: 12h0m0s

Metrics-generator is disabled by default and can be activated by configuring the following values:

metricsGenerator:
  enabled: true
  config:
    storage:
      remote_write:
      - url: http://cortex/api/v1/push
        send_exemplars: true
    #   headers:
    #     x-scope-orgid: operations
# Global overrides
global_overrides:
  defaults:
    metrics_generator:
      processors:
        - service-graphs
        - span-metrics

• Volumes are mounted to /var/tempo. The various directories Tempo needs should be configured as subdirectories (e. g. /var/tempo/wal, /var/tempo/traces). Tempo will create the directories automatically.
• The config file is mounted to /conf/tempo-query.yaml and passed as CLI arg.

storage:
  trace:
    backend: s3
    s3:
      access_key: tempo
      bucket: <your-s3-bucket>
      endpoint: s3.amazonaws.com
      secret_key: <your-secret>
    wal:
      path: /var/tempo/wal

ingest:
  kafka:
    address: <kafka-broker>:9092
    topic: tempo-traces

blockBuilder:
  enabled: true
  replicas: 3  # must equal Kafka partition count

liveStore:
  enabled: true
  replicas: 3  # must equal Kafka partition count

backendScheduler:
  enabled: true

traces:
  otlp:
    http:
      enabled: true
    grpc:
      enabled: true

By default, the chart deploys a single shared memcached StatefulSet (memcached) used for all cache roles — bloom filters, parquet footer, and frontend search. This is the simplest setup and works well for most deployments.

memcached:
  enabled: true

All cache roles (bloom, parquet footer, frontend search) point at the same <release>-memcached service.

You can deploy a dedicated memcached cluster for a specific role by enabling the corresponding per-role section. The shared memcached cluster remains active for the other roles.

For example, to give bloom filters their own cluster while keeping the rest on the shared one:

memcachedBloom:
  enabled: true
  replicas: 2

Available per-role sections:

To run a dedicated memcached cluster for every cache role, disable the shared cluster and enable all three per-role clusters:

memcached:
  enabled: false

memcachedBloom:
  enabled: true
  replicas: 2

memcachedParquetFooter:
  enabled: true
  replicas: 2

memcachedFrontendSearch:
  enabled: true
  replicas: 2

Each role gets its own StatefulSet and Service, and Tempo is configured to use the matching host for every cache type.

gRPC for Open Telemetry is disabled by default, simply flip the bool in the traces block to turn it on.

If you have enabled the gateway as well, this will let you push traces using the default Open Telemetry API path (/opentelemetry.proto.collector.trace.v1.TraceService/Export), on the 4317 port. This port can be overwritten as well in the values.

traces:
  otlp:
    http:
      # -- Enable Tempo to ingest Open Telemetry HTTP traces
      enabled: false
      # -- HTTP receiver advanced config
      receiverConfig: {}
    grpc:
      # -- Enable Tempo to ingest Open Telemetry GRPC traces
      enabled: true
      # -- GRPC receiver advanced config
      receiverConfig: {}
      # -- Default OTLP gRPC port
      port: 4317