Add support for scope in cost-attribution usage tracker #5646

electron0zero · 2025-09-12T12:40:06Z

What this PR does:

Add support for scopes in the cost attribution dimensions attribute matching for cost attribution usage tracker.

Users can now configure dimensions with resource. or span. to only look for the attribute at span or resource level.

With scoped configuration, you can configure tempo to look for values the requested scope and avoid unwanted overwrite from span attributes when the same attribute exists at both resource and span levels.

This change is backwards compatible, and if you need to look at both resource and span scope, you can keep using upscoped attribute in the configuration and it will work as it used to work before this change.

please refer to updated documentation to see more details.

note for reviewers: CHANGELOG.md has few formatting fixes where extra whitespace is cleaned up by my IDE, you can clean the diff with Hide whitespace changes in the diff view.

Benchmarks

Micro benchmarks show almost no change in perf and allocations.

BenchmarkUsageTrackerCollect

$ benchstat Collect-main.txt Collect-branch.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor/usage
cpu: Apple M3 Pro
                       │ Collect-main.txt │         Collect-branch.txt         │
                       │      sec/op      │   sec/op     vs base               │
UsageTrackerCollect-11        8.460µ ± 1%   8.591µ ± 1%  +1.55% (p=0.000 n=10)

                       │ Collect-main.txt │         Collect-branch.txt          │
                       │       B/op       │     B/op      vs base               │
UsageTrackerCollect-11       34.04Ki ± 0%   34.05Ki ± 0%  +0.03% (p=0.030 n=10)

                       │ Collect-main.txt │       Collect-branch.txt       │
                       │    allocs/op     │ allocs/op   vs base            │
UsageTrackerCollect-11         46.00 ± 0%   46.00 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

$ cat Collect-main.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor/usage
cpu: Apple M3 Pro
BenchmarkUsageTrackerCollect-11    	 1428439	      8405 ns/op	   34870 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1416528	      8480 ns/op	   34866 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1407066	      8540 ns/op	   34860 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1413918	      8407 ns/op	   34859 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1429488	      8431 ns/op	   34855 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1422626	      8499 ns/op	   34854 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1412239	      8436 ns/op	   34853 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1429485	      8440 ns/op	   34850 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1420756	      8496 ns/op	   34851 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1408904	      8508 ns/op	   34849 B/op	      46 allocs/op
PASS
ok  	github.com/grafana/tempo/modules/distributor/usage	205.960s

$ cat Collect-branch.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor/usage
cpu: Apple M3 Pro
BenchmarkUsageTrackerCollect-11    	 1409739	      8464 ns/op	   34872 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1410174	      8586 ns/op	   34870 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1388434	      8578 ns/op	   34868 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1390281	      8633 ns/op	   34869 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1384576	      8658 ns/op	   34867 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1380625	      8768 ns/op	   34865 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1391090	      8682 ns/op	   34861 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1395918	      8592 ns/op	   34858 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1399555	      8590 ns/op	   34854 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1397942	      8504 ns/op	   34855 B/op	      46 allocs/op
PASS
ok  	github.com/grafana/tempo/modules/distributor/usage	207.362s

BenchmarkUsageTrackerObserve

$ benchstat Collect-main.txt Collect-branch.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor/usage
cpu: Apple M3 Pro
                       │ Observe-main.txt │       Observe-branch.txt       │
                       │      sec/op      │    sec/op     vs base          │
UsageTrackerObserve-11       5.548µ ± 18%   5.914µ ± 21%  ~ (p=0.529 n=10)

                       │ Observe-main.txt │       Observe-branch.txt       │
                       │       B/op       │    B/op     vs base            │
UsageTrackerObserve-11         0.000 ± 0%   0.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

                       │ Observe-main.txt │       Observe-branch.txt       │
                       │    allocs/op     │ allocs/op   vs base            │
UsageTrackerObserve-11         0.000 ± 0%   0.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

$ cat Collect-main.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor/usage
cpu: Apple M3 Pro
BenchmarkUsageTrackerCollect-11    	 1428439	      8405 ns/op	   34870 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1416528	      8480 ns/op	   34866 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1407066	      8540 ns/op	   34860 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1413918	      8407 ns/op	   34859 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1429488	      8431 ns/op	   34855 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1422626	      8499 ns/op	   34854 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1412239	      8436 ns/op	   34853 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1429485	      8440 ns/op	   34850 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1420756	      8496 ns/op	   34851 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1408904	      8508 ns/op	   34849 B/op	      46 allocs/op
PASS
ok  	github.com/grafana/tempo/modules/distributor/usage	205.960s

$ cat Collect-branch.txt
goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor/usage
cpu: Apple M3 Pro
BenchmarkUsageTrackerCollect-11    	 1409739	      8464 ns/op	   34872 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1410174	      8586 ns/op	   34870 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1388434	      8578 ns/op	   34868 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1390281	      8633 ns/op	   34869 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1384576	      8658 ns/op	   34867 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1380625	      8768 ns/op	   34865 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1391090	      8682 ns/op	   34861 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1395918	      8592 ns/op	   34858 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1399555	      8590 ns/op	   34854 B/op	      46 allocs/op
BenchmarkUsageTrackerCollect-11    	 1397942	      8504 ns/op	   34855 B/op	      46 allocs/op
PASS
ok  	github.com/grafana/tempo/modules/distributor/usage	207.362s

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

modules/distributor/usage/tracker.go

knylander-grafana

Thank you for updating the docs. I have a few minor suggestions to bring your doc in line with our style guide. Overall, great job! I'm going to approve the PR since I'm out of town and don't want to block your work.

CHANGELOG.md

docs/sources/tempo/configuration/usage-tracker.md

modules/distributor/usage/tracker.go

…grafana/tempo into feature/add-option-to-run-once-and-exit * 'feature/add-option-to-run-once-and-exit' of github.com:grafana/tempo: [tools] repalce missing golangci dependency with fork (#5653) feat: don't enqueue records to kafka when the context has been cancelled (#5499) Gracefully handle unsupported block encodings (#5543) vParquet5: dedicated columns for int values (#5639) Add support for scope in cost-attribution usage tracker (#5646) chore: refactor method to wait for a healthy broker (#5618) Update local docker-compose example to scrape /usage_metrics (#5643) Fix race condition in block-builder test (#5645) Upgrade parquet-go (#5644) Cleanup blocks on startup (#5642) Move tracing install to pkg/tracing (#5634)

electron0zero added 5 commits September 12, 2025 20:50

usage: add scope support into cost-attribution usage tracker

08da495

update test

4aeb9b7

update local example config

f9976b1

update tests and add some more

0c7d0a4

Update documention page

7ee61b3

electron0zero force-pushed the scope_in_cost_attr branch from e2703c3 to 7ee61b3 Compare September 12, 2025 17:41

electron0zero changed the title ~~usage: add scope support into cost-attribution usage tracker~~ Sep 12, 2025

electron0zero marked this pull request as ready for review September 12, 2025 17:55

electron0zero requested review from carles-grafana, ie-pham, javiermolinar, joe-elliott, knylander-grafana, mapno, mdisibio, ruslan-mikhailov, stoewer, yvrhdn and zalegrala as code owners September 12, 2025 17:55

Update CHANGELOG.md

b16dd8e

mdisibio reviewed Sep 12, 2025

View reviewed changes

modules/distributor/usage/tracker.go Outdated Show resolved Hide resolved

mdisibio reviewed Sep 12, 2025

View reviewed changes

modules/distributor/usage/tracker.go Outdated Show resolved Hide resolved

knylander-grafana approved these changes Sep 14, 2025

View reviewed changes

electron0zero added 2 commits September 15, 2025 15:36

Address doc review comments

46b8043

replace strings with int8 in the hot path

20a1e1b

electron0zero requested a review from mdisibio September 15, 2025 10:27

electron0zero added 2 commits September 15, 2025 16:59

remove flake from the test

ebcc53f

drop the scope prefix from the metric labels

56adc9b

mdisibio approved these changes Sep 15, 2025

View reviewed changes

modules/distributor/usage/tracker.go Outdated Show resolved Hide resolved

electron0zero merged commit d7c5dc5 into grafana:main Sep 15, 2025
38 of 39 checks passed

electron0zero mentioned this pull request Sep 26, 2025

fix changelog electron0zero/tempo#6

Closed

electron0zero mentioned this pull request Oct 24, 2025

fix docs electron0zero/tempo#7

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for scope in cost-attribution usage tracker #5646

Add support for scope in cost-attribution usage tracker #5646

Uh oh!

electron0zero commented Sep 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

knylander-grafana left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

3 participants

Add support for scope in cost-attribution usage tracker #5646

Add support for scope in cost-attribution usage tracker #5646

Uh oh!

Conversation

electron0zero commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

knylander-grafana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

3 participants

electron0zero commented Sep 12, 2025 •

edited

Loading