tempo-cli: add migrate config command for 2.x to 3.0 migration - #6982
Conversation
Query all blocks regardless of replication factor for non-metrics paths (search, tags, trace-by-ID). Metrics queries continue to filter RF1-only. The rf1_after config field is kept but deprecated for YAML backward compatibility. Proto fields are marked deprecated for wire compatibility. Fixes grafana#6960
9830218 to
9b970d6
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds a new tempo-cli migrate config command to help migrate Tempo 2.x YAML configs to a Tempo 3.0-compatible config, and it removes/deprecates the rf1_after / RF1After filtering behavior across HTTP/API, frontend sharding, and backend block selection so non-metrics query paths now query all blocks regardless of replication factor.
Changes:
- Add
tempo-cli migrate configimplementation, test suite, and YAML fixtures. - Deprecate/ignore
query_frontend.rf1_afterand stop parsing/propagatingrf1Afteracross HTTP APIs, frontend sharders, and tempodb searches. - Update docs/examples/integration configs and tests to reflect the new “query all blocks” behavior.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tempodb/tempodb.go | Removes RF1After-based block filtering in backend searches. |
| tempodb/tempodb_test.go | Updates unit tests for new includeBlock/includeCompactedBlock signatures. |
| tempodb/encoding/common/interfaces.go | Removes RF1After from SearchOptions. |
| pkg/tempopb/tempo.proto | Marks RF1After fields deprecated in protobuf requests. |
| pkg/tempopb/tempo.pb.go | Regenerates protobuf Go code reflecting deprecated RF1After. |
| pkg/api/http.go | Removes rf1After query param support and updates request validation/building. |
| pkg/api/http_test.go | Updates validation tests for the new return signature (no rf1After). |
| pkg/api/search_tags.go | Removes rf1After parsing from tag/tag-values request parsing. |
| modules/querier/querier.go | Stops copying RF1After into backend search options for TraceByID. |
| modules/querier/http.go | Stops accepting/logging/propagating rf1After in TraceByID handlers. |
| modules/frontend/util.go | Replaces RF-based block filter with acceptAllBlocks. |
| modules/frontend/search_sharder.go | Removes RF1After-based filtering of blocks for search sharding. |
| modules/frontend/search_sharder_test.go | Removes RF1After-specific sharder test. |
| modules/frontend/tag_sharder.go | Removes RF1After logic from tag/tag-values sharding. |
| modules/frontend/tag_sharder_test.go | Removes RF1After method from fake request used in tests. |
| modules/frontend/traceid_sharder.go | Stops propagating rf1After into sharded TraceByID requests. |
| modules/frontend/traceid_handlers.go | Updates request validation call signature (no rf1After). |
| modules/frontend/search_handlers_test.go | Updates expected job/block metrics due to querying all blocks. |
| modules/frontend/frontend.go | Removes RF1After propagation into sharder config; updates filterFn comment. |
| modules/frontend/config.go | Marks rf1_after config as deprecated/ignored (omitempty + comment). |
| cmd/tempo/app/config.go | Adds startup warning when query_frontend.rf1_after is set. |
| cmd/tempo-vulture/main.go | Removes vulture rhythm-rf1-after flag and query param injection. |
| integration/util/config-base.yaml | Removes query_frontend.rf1_after from integration base config. |
| example/docker-compose/*/tempo.yaml | Removes query_frontend.rf1_after from example configs. |
| docs/sources/tempo/configuration/manifest.md | Removes rf1_after from the generated manifest snippet. |
| cmd/tempo-cli/main.go | Wires new tempo-cli migrate config subcommand. |
| cmd/tempo-cli/cmd-migrate-config.go | Implements the new migrate-config command. |
| cmd/tempo-cli/cmd-migrate-config_test.go | Adds unit + end-to-end style tests for migrate-config logic. |
| cmd/tempo-cli/test-data/migrate-config/*.yaml | Adds input fixtures for migrate-config tests. |
| CHANGELOG.md | Adds a changelog entry related to rf1_after deprecation. |
9b970d6 to
029920c
Compare
|
This is great! We'll definitely need to mention this in the docs. |
Adds a new CLI command that takes a Tempo 2.x configuration file and
outputs a valid 3.0 configuration, working at the map[string]interface{}
level to preserve unknown fields and env var references.
The command detects deployment mode (monolithic vs microservices),
removes 2.x-only blocks (ingester, compactor, etc.), adds Kafka ingest
config for microservices mode, sets compaction_disabled in overrides,
and strips the removed local-blocks metrics generator processor.
029920c to
9f48348
Compare
There was a problem hiding this comment.
Pull request overview
Adds a new tempo-cli migrate config command to help users migrate Tempo 2.x config files to a Tempo 3.0-compatible configuration, alongside docs, tests, and changelog updates.
Changes:
- Introduces
tempo-cli migrate configto transform 2.x YAML configs (remove obsolete blocks, add ingest Kafka config in microservices mode, adjust overrides, strip local-blocks processor). - Adds end-to-end and unit tests plus YAML fixtures covering common migration scenarios (env vars, legacy overrides detection, local-blocks cleanup).
- Updates docs (upgrade guide + tempo-cli reference) and adds a changelog entry for the new CLI command.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md | Links the upgrade guide to the new tempo-cli migrate config command. |
| docs/sources/tempo/operations/tempo_cli.md | Documents the new migrate config command, flags, and caveats. |
| cmd/tempo-cli/test-data/migrate-config/with-per-tenant-override-config-input.yaml | Fixture covering per_tenant_override_config warning behavior. |
| cmd/tempo-cli/test-data/migrate-config/with-local-blocks-input.yaml | Fixture covering removal of local_blocks config + local-blocks processor entry. |
| cmd/tempo-cli/test-data/migrate-config/with-env-vars-input.yaml | Fixture covering ${VAR} preservation / validation skip behavior. |
| cmd/tempo-cli/test-data/migrate-config/no-target-input.yaml | Fixture covering default mode detection when target is absent. |
| cmd/tempo-cli/test-data/migrate-config/monolithic-basic-input.yaml | Fixture for basic monolithic migration inputs. |
| cmd/tempo-cli/test-data/migrate-config/microservices-basic-input.yaml | Fixture for basic microservices migration inputs (Kafka required). |
| cmd/tempo-cli/test-data/migrate-config/legacy-overrides-input.yaml | Fixture for legacy overrides detection erroring out. |
| cmd/tempo-cli/main.go | Wires the new migrate config subcommand into tempo-cli. |
| cmd/tempo-cli/cmd-migrate-config_test.go | Adds unit + end-to-end tests for config migration helpers. |
| cmd/tempo-cli/cmd-migrate-config.go | Implements the migrate config command and helper functions. |
| CHANGELOG.md | Adds a [FEATURE] entry for the new command. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new tempo-cli migrate config subcommand intended to help operators migrate Tempo 2.x YAML configs to a Tempo 3.0-compatible config, and documents it in the upgrade guide and CLI docs.
Changes:
- Introduces
tempo-cli migrate configcommand that rewrites a 2.x config at the YAML-map level (drop removed blocks, addingest.kafkain microservices mode, adjust overrides, remove local-blocks processor) and then validates against Tempo’s 3.0 config. - Adds end-to-end and unit tests plus YAML fixtures for common migration scenarios (env vars, local-blocks, legacy overrides, etc.).
- Updates docs (upgrade guide + tempo-cli docs) and adds a changelog entry for the new command.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md | Mentions the new CLI command in the 3.0 upgrade breaking-changes list. |
| docs/sources/tempo/operations/tempo_cli.md | Documents tempo-cli migrate config usage, behavior, and warnings. |
| cmd/tempo-cli/main.go | Wires the new migrate config command into the CLI. |
| cmd/tempo-cli/cmd-migrate-config.go | Implements config migration + validation for 2.x → 3.0. |
| cmd/tempo-cli/cmd-migrate-config_test.go | Unit + end-to-end tests for migration behavior. |
| cmd/tempo-cli/test-data/migrate-config/*.yaml | Migration fixtures for test coverage. |
| CHANGELOG.md | Adds release note entry for the new CLI command. |
There was a problem hiding this comment.
Pull request overview
Adds a new tempo-cli migrate config command intended to automate much of the Tempo 2.x → 3.0 configuration migration workflow, plus accompanying docs, test coverage, and changelog entry.
Changes:
- Introduces
tempo-cli migrate configto transform a 2.x YAML config into a 3.0-compatible config map (remove obsolete sections, addingest.kafkain microservices mode, adjust overrides, removelocal-blocksMG config). - Adds end-to-end and unit tests plus YAML fixtures for common migration scenarios (env vars, legacy overrides, local-blocks, etc.).
- Documents the command and references it from the 3.0 upgrade guide; adds a changelog entry.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md | Mentions using tempo-cli migrate config during 3.0 upgrade and clarifies ingest.kafka expectations. |
| docs/sources/tempo/operations/tempo_cli.md | Adds documentation section for the new migrate config command. |
| cmd/tempo-cli/test-data/migrate-config/with-per-tenant-override-config-input.yaml | Fixture for external per-tenant override config warning path. |
| cmd/tempo-cli/test-data/migrate-config/with-local-blocks-input.yaml | Fixture covering local_blocks removal and processor list cleanup. |
| cmd/tempo-cli/test-data/migrate-config/with-env-vars-input.yaml | Fixture ensuring ${VAR} placeholders survive migration. |
| cmd/tempo-cli/test-data/migrate-config/no-target-input.yaml | Fixture for defaulting to monolithic when target is absent. |
| cmd/tempo-cli/test-data/migrate-config/monolithic-basic-input.yaml | Fixture for basic monolithic migration behavior. |
| cmd/tempo-cli/test-data/migrate-config/microservices-basic-input.yaml | Fixture for microservices mode and required Kafka flags. |
| cmd/tempo-cli/test-data/migrate-config/legacy-overrides-input.yaml | Fixture ensuring legacy overrides format is rejected. |
| cmd/tempo-cli/main.go | Wires the new subcommand into the CLI. |
| cmd/tempo-cli/cmd-migrate-config_test.go | Adds unit + end-to-end tests for migration helpers and behavior. |
| cmd/tempo-cli/cmd-migrate-config.go | Implements the migration command and helper functions. |
| CHANGELOG.md | Adds a [FEATURE] entry for the new command. |
1c8a3fa to
c0d21f9
Compare
| tag := t.Field(i).Tag.Get("yaml") | ||
| if tag == "" || tag == "-" || tag == ",inline" { | ||
| continue | ||
| } | ||
| name, _, _ := strings.Cut(tag, ",") | ||
| if name != "" { | ||
| fields[name] = struct{}{} | ||
| } |
There was a problem hiding this comment.
nit: "" and ",inline" are already covered by name != "" after split
There was a problem hiding this comment.
Good catch. Simplified to just check tag == "-" and rely on name != "" from the split.
| ovrMap, ok := overridesRaw.(map[string]interface{}) | ||
| if !ok { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Does it make sense to add a warning that format is not correct?
There was a problem hiding this comment.
Same question to other casts like this, e.g. https://github.com/grafana/tempo/pull/6982/changes#diff-74a37b96ee920fd023c5806ceb7cfe29f4059eeb27b3d22d8ab94b1f58222716R204
There was a problem hiding this comment.
Addressed. Added an asMap helper and warn when a per-tenant override entry is not a map (the main case where this could indicate malformed user input). Also replaced the repeated raw, ok := m[k]; if ok { v, ok := raw.(map[string]any); if ok { ... } } patterns with a shared extractNestedMap helper — see the other reply thread.
| } | ||
|
|
||
| // 2. Detect deployment mode | ||
| mode := detectMode(m, cmd.Mode, &warnings) |
There was a problem hiding this comment.
Nit: I think having a structure with methods instead of functions, that can store shared vars like warning list would simplify code a bit
There was a problem hiding this comment.
I opted to keep the plain functions for now since each one takes the root map, but I did reduce repetition by factoring out extractNestedMap, asMap, and consolidating removeLocalBlocksProcessorConfig. Happy to revisit the struct+methods approach in a follow-up if you feel strongly — the current structure still shares state via the warnings *[]string param.
| mgRaw, ok := m[mgKey] | ||
| if !ok { | ||
| return | ||
| } | ||
| mg, ok := mgRaw.(map[string]interface{}) | ||
| if !ok { | ||
| return | ||
| } | ||
| procRaw, ok := mg["processor"] | ||
| if !ok { | ||
| return | ||
| } | ||
| proc, ok := procRaw.(map[string]interface{}) | ||
| if !ok { | ||
| return | ||
| } |
There was a problem hiding this comment.
Nit. This pattern is repetitive in the script.
Proposal for simplification:
func extractAsMap(m map[string]interface{}, nestedKeys []string) (map[string]interface{}, bool) {
for _, key := range nestedKeys {
v, ok := m[key]
if !ok {
return nil, ok
}
m, ok = v.(map[string]interface{})
if !ok {
return nil, ok
}
}
return m, true
}There was a problem hiding this comment.
Good suggestion. Added extractNestedMap(m, path...) (map[string]any, bool) which walks a key path and returns the deepest map. Used it in detectLegacyOverrides, cleanLocalBlocks, removeLocalBlocksProcessorConfig, and removeLocalBlocksFromProcessorList. Eliminated ~30 lines of repetitive cast code.
| // validTargets returns the list of valid Tempo 3.0 target values. | ||
| func validTargets() []string { | ||
| return []string{ |
There was a problem hiding this comment.
nit: can be just var []string or map[string]bool :)
There was a problem hiding this comment.
Done. Changed yamlFieldNames to return map[string]bool (cleaner membership check with m[k] instead of _, ok := m[k]), and converted validTargets from a function returning []string to a package-level map[string]bool for O(1) lookup.
| @@ -0,0 +1,14 @@ | |||
| target: all | |||
There was a problem hiding this comment.
For these test files, it would be also nice to have*-output.yaml and test that checks that output of an -input files matches it. These files will give clear expectation on how the script works and will be easier to check.
There was a problem hiding this comment.
Done. Added *-expected.yaml files for each input and rewrote TestMigrateConfigEndToEnd to run the full command pipeline (including validation) and compare stdout byte-for-byte against the expected file. Covered inputs: monolithic, microservices, no-target, with-local-blocks, with-per-tenant-override-config, with-env-vars, and scalable-single-binary. Error cases still assert on error strings.
# Conflicts: # docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md
There was a problem hiding this comment.
Pull request overview
Adds a new tempo-cli migrate config command to help users migrate Tempo 2.x YAML configs to a Tempo 3.0-compatible config, plus accompanying docs, tests, and changelog entry.
Changes:
- Introduces
tempo-cli migrate configwith YAML-map-based rewriting (drop removed top-level blocks, addingest.kafkain microservices mode, adjust overrides, remove local-blocks processor config) and strict validation. - Adds docs describing the new command and updates the upgrade guide to reference it.
- Adds test data + unit tests for migration helper behavior, and a changelog entry.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md | Updates 3.0 upgrade notes and links to the new migrate-config command. |
| docs/sources/tempo/operations/tempo_cli.md | Documents the new tempo-cli migrate config command, flags, and caveats. |
| cmd/tempo-cli/test-data/migrate-config/*.yaml | Adds input fixtures for migration scenarios (monolithic/microservices, env vars, legacy overrides, etc.). |
| cmd/tempo-cli/main.go | Registers the new migrate config subcommand. |
| cmd/tempo-cli/cmd-migrate-config_test.go | Adds unit tests for migration helper functions and end-to-end map transformation. |
| cmd/tempo-cli/cmd-migrate-config.go | Implements the migrate-config command + helpers and validation. |
| CHANGELOG.md | Adds a release note entry for the new CLI command. |
…ty target rewrite
|
@mapno Will this work for single binary? |
knylander-grafana
left a comment
knylander-grafana
left a comment
There was a problem hiding this comment.
Approving the docs. Well done -- and delighted to see that both microservices and monolithic are included in the doc update.
The main suggestion I have is spelling out configuration instead of using config.
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new tempo-cli migrate config command to help operators migrate Tempo 2.x YAML configs to a Tempo 3.0-compatible configuration, and documents the workflow as part of the 2.x→3.0 upgrade path.
Changes:
- Introduces
tempo-cli migrate configthat rewrites/removes deprecated 2.x sections, injectsingest.kafkafor microservices mode, adjusts overrides to disable compaction during parallel-run migration, and removeslocal_blocks/local-blocksmetrics-generator settings. - Adds end-to-end and unit tests plus YAML fixtures for common migration scenarios (env vars, SSB target rewrite, legacy overrides detection, etc.).
- Updates upgrade docs, CLI docs, and the changelog to reference the new migration command.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/sources/tempo/set-up-for-tracing/setup-tempo/upgrade.md | Updates 3.0 upgrade notes to mention config migration + the new CLI command. |
| docs/sources/tempo/operations/tempo_cli.md | Documents tempo-cli migrate config usage, caveats, and examples. |
| cmd/tempo-cli/main.go | Registers the new migrate config subcommand. |
| cmd/tempo-cli/cmd-migrate-config.go | Implements the config migration pipeline, warnings, and validation. |
| cmd/tempo-cli/cmd-migrate-config_test.go | Adds unit + end-to-end tests for migration behaviors. |
| cmd/tempo-cli/test-data/migrate-config/*.yaml | Test fixtures for migration inputs/expected outputs. |
| CHANGELOG.md | Adds a release note entry for the new CLI command. |
|
|
||
| Migrate a Tempo 2.x configuration file to a valid 3.0 configuration. The command removes obsolete configuration sections (such as `ingester`, `ingester_client`, and `compactor`), adds Kafka ingest configuration for microservices mode, disables compaction in overrides for parallel operation during migration, and strips the removed `local-blocks` metrics-generator processor. | ||
|
|
||
| The tool works at the YAML map level rather than rewriting the file from fully decoded Tempo structs, so environment variable references like `${VAR}` are preserved. |
…na#6982) * Remove rf1_after from query frontend and query paths Query all blocks regardless of replication factor for non-metrics paths (search, tags, trace-by-ID). Metrics queries continue to filter RF1-only. The rf1_after config field is kept but deprecated for YAML backward compatibility. Proto fields are marked deprecated for wire compatibility. Fixes grafana#6960 * fmt * Add tempo-cli migrate config command for 2.x to 3.0 migration Adds a new CLI command that takes a Tempo 2.x configuration file and outputs a valid 3.0 configuration, working at the map[string]interface{} level to preserve unknown fields and env var references. The command detects deployment mode (monolithic vs microservices), removes 2.x-only blocks (ingester, compactor, etc.), adds Kafka ingest config for microservices mode, sets compaction_disabled in overrides, and strips the removed local-blocks metrics generator processor. * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Handle SSB * -) * ups * Fix test * Address review feedback: extractNestedMap helper, snapshot tests, empty target rewrite * Apply suggestions from code review Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
What this PR does:
Adds a
tempo-cli migrate configcommand that takes a Tempo 2.x configuration file and outputs a valid 3.0 configuration. This eliminates the most error-prone step of the migration — manually editing config files.The command migrates the config by:
ingester,ingester_client,compactor,metrics_generator_clientingest.kafka.{address,topic}local_blocksprocessor config andlocal-blocksfrom processors listsoverrides.defaults.compaction.compaction_disabled: true(part of the migration)Which issue(s) this PR fixes:
Fixes #6979
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]