chore: remove remaining app ingester config by javiermolinar · Pull Request #6667 · grafana/tempo

javiermolinar · 2026-03-12T14:58:30Z

What this PR does:

This PR removes the ingester module to loaded. It also remove the deprecated code in the distributor.

This is a breaking change and we need to patch any place where the ingester config is still in use

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

javiermolinar · 2026-03-12T15:02:07Z

@codex review

chatgpt-codex-connector · 2026-03-12T15:03:06Z

Codex couldn't complete this request. Try again later.

javiermolinar · 2026-03-13T10:58:30Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28fb90eab1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-13T11:11:01Z

 func (c *Config) CheckConfig() []ConfigWarning {
 	var warnings []ConfigWarning
-	if c.Ingester.CompleteBlockTimeout < c.StorageConfig.Trace.BlocklistPoll {
-		warnings = append(warnings, warnCompleteBlockTimeout)
-	}
-
 	if c.BackendWorker.Compactor.BlockRetention < c.StorageConfig.Trace.BlocklistPoll {
 		warnings = append(warnings, warnBlockRetention)


Warn when live-store timeout is below blocklist poll

CheckConfig() no longer emits any warning for complete_block_timeout versus storage.trace.blocklist_poll, but the same stale-blocklist gap still exists with the live-store path (flushed data is evicted after live_store.complete_block_timeout while queriers can still be on an older blocklist), so operators can now ship a misconfiguration that causes transient 404s without any startup warning; this check should be retained against the live-store setting instead of being removed entirely.

Useful? React with 👍 / 👎.

Very good catch

ruslan-mikhailov · 2026-03-17T09:48:36Z

+

 ### 3.0 Cleanup
+* [CHANGE] **BREAKING CHANGE** Remove remaining app ingester config [#6667](https://github.com/grafana/tempo/pull/6667) (@javiermolinar)


Should be after the blank line

ruslan-mikhailov · 2026-03-17T09:51:09Z

 	"github.com/grafana/tempo/modules/frontend"
 	"github.com/grafana/tempo/modules/generator"
-	"github.com/grafana/tempo/modules/ingester"
-	ingester_client "github.com/grafana/tempo/modules/ingester/client"


No calls to this package left. What do you think about removing it too?

If we delete it, we can delete tempopb.PusherClient I think

Wow nice finding. But I want do it in a different pr since it involves delete a lot of code

Copilot

Pull request overview

This PR completes the removal of the legacy ingester configuration surface area and ingester write-path code, updating Tempo’s wiring, dashboards, examples, and documentation to align with the live-store + Kafka-based write path.

Changes:

Removes distributor → ingester write path code/config and related metrics, along with associated unit tests and validation.
Updates Tempo mixin dashboards/config and multiple deployment examples to replace ingester references with live-store.
Updates generated configuration manifest and docs to reflect removal of ingester/ingester-client configuration.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/docsgen/generate_manifest.go	Stops overriding ingester lifecycler defaults; ensures live-store defaults are used in generated manifest.
operations/tempo-mixin/dashboards/tempo-writes.libsonnet	Removes panel query for distributor→ingester append failures.
operations/tempo-mixin/dashboards/rollout-progress.libsonnet	Updates rollout service regex to use `live-store` instead of `ingester`.
operations/tempo-mixin/config.libsonnet	Removes `ingester` job matcher from mixin config.
operations/tempo-mixin-compiled/dashboards/tempo-writes.json	Compiled dashboard update matching the libsonnet change.
operations/tempo-mixin-compiled/dashboards/tempo-rollout-progress.json	Compiled dashboard update matching the libsonnet change.
modules/distributor/receiver/shim.go	Updates metric help text to reflect distributor routing semantics.
modules/distributor/distributor_test.go	Removes ingester write-path tests/mocks and adapts helpers to new distributor constructor.
modules/distributor/distributor.go	Removes ingester write path (pool/ring/client) and switches Kafka timeout source.
modules/distributor/config.go	Removes deprecated ingester/kafka write-path toggles and ingester client factory hooks.
integration/util/config-base.yaml	Removes deprecated distributor toggles and ingester_client timeout block.
example/tk/tempo-microservices/main.jsonnet	Removes ingester component config overrides; keeps live-store.
example/tk/readme.md	Updates override example from `ingester+` to `live_store+`.
example/nomad/tempo-distributed/tempo.hcl	Replaces ingester group with live-store and normalizes file formatting.
example/nomad/tempo-distributed/config.yml	Replaces ingester config with live-store + ingest config for Kafka write path.
example/docker-compose/single-binary/tempo.yaml	Removes deprecated distributor write-path toggles.
example/docker-compose/multitenant/tempo.yaml	Removes deprecated distributor write-path toggles.
example/docker-compose/distributed/tempo.yaml	Removes deprecated distributor write-path toggles.
example/docker-compose/debug/tempo.yaml	Removes deprecated distributor write-path toggles.
docs/sources/tempo/configuration/network/tls.md	Updates TLS client config reference to `live_store_client`.
docs/sources/tempo/configuration/manifest.md	Removes distributor ingester/kafka toggles and removes ingester/ingester-client sections from manifest.
docs/sources/tempo/configuration/_index.md	Updates configuration documentation language and client config references to live-store.
cmd/tempo/main.go	Removes single-binary ingester forced ring config; keeps single-binary live-store/generator ring forcing.
cmd/tempo/app/overrides_validation_test.go	Removes ingester replication-factor based override validation tests.
cmd/tempo/app/overrides_validation.go	Removes ingester replication-factor based override validation logic.
cmd/tempo/app/modules.go	Updates distributor module initialization to new constructor signature.
cmd/tempo/app/config_test.go	Updates config warning test to set `LiveStore.CompleteBlockTimeout`.
cmd/tempo/app/config.go	Removes ingester/ingester-client from root config and updates warning messaging to live-store.
cmd/tempo/app/app.go	Removes ingester readiness check from `/ready`.
CHANGELOG.md	Adds breaking-change entry for ingester config removal (but currently introduces a duplicate entry elsewhere).

Comments suppressed due to low confidence (1)

modules/distributor/config.go:94

KafkaConfig defaults (including write_timeout) aren't applied because RegisterFlagsAndApplyDefaults never calls cfg.KafkaConfig.RegisterFlags.... This can leave KafkaConfig.WriteTimeout at 0, causing context.WithTimeout(..., 0) in sendToKafka() to immediately time out (and the flags for distributor.kafka_config also won't be registered). Call cfg.KafkaConfig.RegisterFlagsWithPrefix(...) (using the distributor's prefix) so defaults are set and flags are available.

// RegisterFlagsAndApplyDefaults registers flags and applies defaults
func (cfg *Config) RegisterFlagsAndApplyDefaults(prefix string, f *flag.FlagSet) {
	flagext.DefaultValues(&cfg.DistributorRing)
	cfg.DistributorRing.KVStore.Store = "memberlist"
	cfg.DistributorRing.HeartbeatTimeout = 5 * time.Minute

	cfg.RetryAfterOnResourceExhausted = 0
	cfg.OverrideRingKey = distributorRingKey

	cfg.MaxAttributeBytes = 2048 // 2KB

	f.BoolVar(&cfg.LogReceivedSpans.Enabled, util.PrefixConfig(prefix, "log-received-spans.enabled"), false, "Enable to log every received span to help debug ingestion or calculate span error distributions using the logs.")
	f.BoolVar(&cfg.LogReceivedSpans.IncludeAllAttributes, util.PrefixConfig(prefix, "log-received-spans.include-attributes"), false, "Enable to include span attributes in the logs.")
	f.BoolVar(&cfg.LogReceivedSpans.FilterByStatusError, util.PrefixConfig(prefix, "log-received-spans.filter-by-status-error"), false, "Enable to filter out spans without status error.")

	f.BoolVar(&cfg.LogDiscardedSpans.Enabled, util.PrefixConfig(prefix, "log-discarded-spans.enabled"), false, "Enable to log every discarded span to help debug ingestion or calculate span error distributions using the logs.")
	f.BoolVar(&cfg.LogDiscardedSpans.IncludeAllAttributes, util.PrefixConfig(prefix, "log-discarded-spans.include-attributes"), false, "Enable to include span attributes in the logs.")
	f.BoolVar(&cfg.LogDiscardedSpans.FilterByStatusError, util.PrefixConfig(prefix, "log-discarded-spans.filter-by-status-error"), false, "Enable to filter out spans without status error.")

	cfg.Usage.RegisterFlagsAndApplyDefaults(prefix, f)
}

You can also share your feedback on Copilot code review. Take the survey.

 * [BUGFIX] fix(traceql): err on division by zero [#6580](https://github.com/grafana/tempo/pull/6580) (@Proximyst)
 * [BUGFIX] Return 400 instead of 500 when query_range or query_instant requests have unparseable start/end parameters [#6694](https://github.com/grafana/tempo/pull/6694) (@ruslan-mikhailov)

+


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR removes remaining ingester-oriented configuration and wiring from the Tempo app and distributor write path, completing the transition to LiveStore/Kafka-based ingestion and updating mixins/examples/docs accordingly.

Changes:

Remove the distributor’s ingester write path (ring/pool/client, metrics, and related tests), leaving Kafka and forwarder paths.
Update Tempo mixin dashboards/config to remove ingester panels/job matchers and rename rollout service matching to live-store.
Update app config, examples, docs, and manifest generation to drop ingester / ingester_client references and use live_store / live_store_client.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/docsgen/generate_manifest.go	Stops overriding ingester-specific fields when generating the config manifest.
operations/tempo-mixin/dashboards/tempo-writes.libsonnet	Removes the ingester-append-failures query from distributor write dashboards.
operations/tempo-mixin/dashboards/rollout-progress.libsonnet	Updates rollout service regex to track `live-store` instead of `ingester`.
operations/tempo-mixin/config.libsonnet	Removes `ingester` from mixin job configuration.
operations/tempo-mixin-compiled/dashboards/tempo-writes.json	Compiled dashboard updated to match ingester-metric removal.
operations/tempo-mixin-compiled/dashboards/tempo-rollout-progress.json	Compiled dashboard updated to match `live-store` regex change.
modules/distributor/receiver/shim.go	Updates distributor push duration metric help text to remove ingester wording.
modules/distributor/distributor_test.go	Removes ingester write-path test scaffolding and updates helpers to the new distributor constructor.
modules/distributor/distributor.go	Removes ingester client/ring/pool write path and related metrics; switches Kafka write timeout source.
modules/distributor/config.go	Removes deprecated ingester/kafka write-path toggles and ingester client factory hook.
integration/util/config-base.yaml	Drops deprecated distributor write-path toggles and ingester_client remote timeout from integration base config.
example/tk/tempo-microservices/main.jsonnet	Removes ingester component config and deprecated distributor write-path toggles from Jsonnet example.
example/tk/readme.md	Updates override example to use `live_store+` instead of `ingester+`.
example/nomad/tempo-distributed/tempo.hcl	Replaces Nomad ingester group with live-store group and normalizes formatting.
example/nomad/tempo-distributed/config.yml	Updates example config from `ingester` to `live_store`, adds ingest/Kafka and live-store partition ring settings.
example/docker-compose/single-binary/tempo.yaml	Removes deprecated distributor write-path toggles.
example/docker-compose/multitenant/tempo.yaml	Removes deprecated distributor write-path toggles.
example/docker-compose/distributed/tempo.yaml	Removes deprecated distributor write-path toggles.
example/docker-compose/debug/tempo.yaml	Removes deprecated distributor write-path toggles.
docs/sources/tempo/configuration/network/tls.md	Updates TLS doc references from `ingester_client` to `live_store_client`.
docs/sources/tempo/configuration/manifest.md	Removes ingester and ingester_client sections from the generated manifest docs.
docs/sources/tempo/configuration/_index.md	Updates configuration docs wording and examples to reflect live-store/write-path terminology.
cmd/tempo/main.go	Removes single-binary forced ingester lifecycler config; keeps/extends single-binary ring forcing for live-store.
cmd/tempo/app/overrides_validation_test.go	Removes overrides validation tests that depended on ingester replication factor.
cmd/tempo/app/overrides_validation.go	Removes runtime override validation that referenced ingester replication factor.
cmd/tempo/app/modules.go	Updates distributor initialization to match new constructor signature (no ingester client/ring).
cmd/tempo/app/config_test.go	Updates config warnings test to use `LiveStore.CompleteBlockTimeout`.
cmd/tempo/app/config.go	Removes ingester/ingester_client from root config; switches warning logic to live-store.
cmd/tempo/app/app.go	Removes ingester readiness check and ingester field from the app struct.
CHANGELOG.md	Adds the breaking-change entry under “3.0 Cleanup” and reorders an existing entry.

You can also share your feedback on Copilot code review. Take the survey.

 * [BUGFIX] fix(traceql): err on division by zero [#6580](https://github.com/grafana/tempo/pull/6580) (@Proximyst)
 * [BUGFIX] Return 400 instead of 500 when query_range or query_instant requests have unparseable start/end parameters [#6694](https://github.com/grafana/tempo/pull/6694) (@ruslan-mikhailov)

+


* distributor: remove ingester write path * docs: remove stale distributor ingester config * mixin: remove stale distributor ingester panel * app: remove ingester module wiring * cmd: remove remaining app ingester config * docs(changelog): add entry for grafana#6667 * manifest * refactor: remove leftover distributor ingester references * chore: remove remaining ingester example references * add warning check * Remove duplicate changelog entry Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix changelog --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

javiermolinar and others added 8 commits March 12, 2026 15:34

distributor: remove ingester write path

949aea3

docs: remove stale distributor ingester config

265fd62

mixin: remove stale distributor ingester panel

7005cf2

app: remove ingester module wiring

8079739

cmd: remove remaining app ingester config

275d949

docs(changelog): add entry for grafana#6667

0beb674

Merge branch 'main' into remove-distributor-ingester-path

72a34cf

manifest

b189648

javiermolinar marked this pull request as ready for review March 12, 2026 15:01

javiermolinar requested review from carles-grafana, electron0zero, ie-pham, knylander-grafana, mapno, mattdurham, mdisibio, oleg-kozlyuk-grafana, ruslan-mikhailov, stoewer, yvrhdn and zalegrala as code owners March 12, 2026 15:02

javiermolinar added 3 commits March 13, 2026 11:10

fix: resolve merge conflicts

6592e1b

refactor: remove leftover distributor ingester references

86154a3

chore: remove remaining ingester example references

28fb90e

chatgpt-codex-connector Bot reviewed Mar 13, 2026

View reviewed changes

add warning check

94b5a08

ruslan-mikhailov reviewed Mar 17, 2026

View reviewed changes

Merge branch 'main' into remove-distributor-ingester-path

731f664

Copilot AI review requested due to automatic review settings March 17, 2026 10:33

Copilot started reviewing on behalf of javiermolinar March 17, 2026 10:33 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

Remove duplicate changelog entry

8baedcc

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 17, 2026 10:45

Copilot started reviewing on behalf of javiermolinar March 17, 2026 10:45 View session

javiermolinar enabled auto-merge (squash) March 17, 2026 10:51

Copilot AI reviewed Mar 17, 2026

View reviewed changes

ruslan-mikhailov approved these changes Mar 17, 2026

View reviewed changes

javiermolinar disabled auto-merge March 17, 2026 10:57

fix changelog

ba21756

javiermolinar merged commit 92be6be into grafana:main Mar 17, 2026
26 checks passed

electron0zero mentioned this pull request Mar 19, 2026

disable lagacy overrides electron0zero/tempo#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: remove remaining app ingester config - #6667

chore: remove remaining app ingester config#6667
javiermolinar merged 15 commits into
grafana:mainfrom
javiermolinar:remove-distributor-ingester-path

javiermolinar commented Mar 12, 2026 •

edited

Loading

javiermolinar commented Mar 12, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Mar 12, 2026

javiermolinar commented Mar 13, 2026

chatgpt-codex-connector Bot left a comment

chatgpt-codex-connector Bot Mar 13, 2026

javiermolinar Mar 13, 2026

ruslan-mikhailov Mar 17, 2026

Uh oh!

ruslan-mikhailov Mar 17, 2026 •

edited

Loading

ruslan-mikhailov Mar 17, 2026

javiermolinar Mar 17, 2026 •

edited

Loading

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Labels

3 participants



		### 3.0 Cleanup
		* [CHANGE] BREAKING CHANGE Remove remaining app ingester config [#6667](https://github.com/grafana/tempo/pull/6667) (@javiermolinar)

		* [BUGFIX] fix(traceql): err on division by zero [#6580](https://github.com/grafana/tempo/pull/6580) (@Proximyst)
		* [BUGFIX] Return 400 instead of 500 when query_range or query_instant requests have unparseable start/end parameters [#6694](https://github.com/grafana/tempo/pull/6694) (@ruslan-mikhailov)

Uh oh!

Conversation

javiermolinar commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

javiermolinar commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

chatgpt-codex-connector Bot commented Mar 12, 2026

javiermolinar commented Mar 13, 2026

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

chatgpt-codex-connector Bot Mar 13, 2026

Choose a reason for hiding this comment

javiermolinar Mar 13, 2026

Choose a reason for hiding this comment

ruslan-mikhailov Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

ruslan-mikhailov Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

ruslan-mikhailov Mar 17, 2026

Choose a reason for hiding this comment

javiermolinar Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Labels

3 participants

javiermolinar commented Mar 12, 2026 •

edited

Loading

javiermolinar commented Mar 12, 2026 •

edited

Loading

ruslan-mikhailov Mar 17, 2026 •

edited

Loading

javiermolinar Mar 17, 2026 •

edited

Loading