Remove MustNewConstMetric and validate config in usage tracker

[electron0zero]

What this PR does:

prometheus.MustNewConstMetric will panic when you try to create an metric with invalid label name. in prometheus world any labels that start with __ are invalid because they are reserved.

registering same labels will also cause prometheus.MustNewConstMetric to panic.

you can configure overrides to map to invalid label names or use label names that are sanitized into __ and it will cause distributors to panic when you scrape /usage_metrics to collect usage tracker metrics.

there are also ways to cause panic by pushing data into tempo that results in an invalid metric.

prometheus.MustNewConstMetric is designed to panic and is expected to panic when you have invalid config.

in this case, we should use prometheus.NewConstMetric (it's what prometheus.MustNewConstMetric uses internally) and handle errors when we can't guarantee valid label names.

This PR replaces usage of prometheus.MustNewConstMetric in usage tracker with prometheus.NewConstMetric so never panic.

Along with that, this PR does few more things:

• validate per tenant runtime overrides and fail if overrides contains a usage tracker config that results in invalid label names (note: this will cause tempo to fail at startup with errors like level=error ts=2025-11-19T18:23:43.440930939Z caller=app.go:231 msg="module failed" module=overrides-api err="failed to start overrides-api, because it depends on module overrides, which has failed: invalid service state: Failed, expected: Running, failure: starting module overrides: invalid service state: Failed, expected: Running, failure: failed to start subservices: not healthy, 0 terminated, 1 failed: [failed to load runtime config: load file: validating overrides for * failed: cost_attribution.dimensions has duplicate label name: 'name', both 'service.name' and 'span.http.target' map to it]"
• validate usage tracker config in user configurable overrides API so we don't allow invalid config into tempo. this will reject al
• log the errors with a rate limited logger from prometheus.NewConstMetric in case of an invalid label name in usage tracker config.
• Add tempo_distributor_usage_tracker_errors_total counter to capture the errors with reason for the failures in the the usage tracker

More Details

I used a config like ⏬ to cause a panic:

    cost_attribution:
      dimensions:
        service.name: "__name__"
        span.http.target: "service_route"

here is panic log, I used local docker-compose in this test

tempo-1       | panic: "__name__" is not a valid label name for metric "tempo_usage_tracker_bytes_received_total"
tempo-1       |
tempo-1       | goroutine 7167 [running]:
tempo-1       | github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
tempo-1       | 	/Users/suraj/wd/grafana/tempo/vendor/github.com/prometheus/client_golang/prometheus/value.go:129
tempo-1       | github.com/grafana/tempo/modules/distributor/usage.(*Tracker).Collect(0x4000e2ff80?, 0x401fc4d340)
tempo-1       | 	/Users/suraj/wd/grafana/tempo/modules/distributor/usage/tracker.go:397 +0x18c
tempo-1       | github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
tempo-1       | 	/Users/suraj/wd/grafana/tempo/vendor/github.com/prometheus/client_golang/prometheus/registry.go:458 +0xac
tempo-1       | created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather in goroutine 7165
tempo-1       | 	/Users/suraj/wd/grafana/tempo/vendor/github.com/prometheus/client_golang/prometheus/registry.go:467 +0x4b4

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[carles-grafana]

this seems redundant, I think essentially we only need to check that it's valid utf-8

[electron0zero]

I think essentially we only need to check that it's valid utf-8

prometheus has support for utf-8 but it's not there yet, so we still need to follow the Legacy validation to support most of the users.

in Legacy validation, prometheus is much more strict and won't allowed reserved labels (anything that starts with __), or something like __name__ so I am using the prometheus's code to do the validation instead of re-implementing the validations.

also, you can't use the same label twice or it will err

[carles-grafana]

My point is that creating the metric here is unnecessary, because looking at the function definition the only relevant validation is UTF-8:

func NewConstMetric(desc *Desc, valueType ValueType, value float64, labelValues ...string) (Metric, error) {
	if desc.err != nil {
		return nil, desc.err
	}
	if err := validateLabelValues(labelValues, len(desc.variableLabels.names)); err != nil {
		return nil, err
	}

	metric := &dto.Metric{}
	if err := populateMetric(valueType, value, MakeLabelPairs(desc, labelValues), nil, metric, nil); err != nil {
		return nil, err
	}

	return &constMetric{
		desc:   desc,
		metric: metric,
	}, nil
}

We only care about validating the label name, and we can see that the only relevant validation is the UTF-8, so we just need to check that and we avoid doing extra work:

func validateLabelValues(vals []string, expectedNumberOfValues int) error {
	if len(vals) != expectedNumberOfValues {
		// The call below makes vals escape, copy them to avoid that.
		vals := append([]string(nil), vals...)
		return fmt.Errorf(
			"%w: expected %d label values but got %d in %#v",
			errInconsistentCardinality, expectedNumberOfValues,
			len(vals), vals,
		)
	}

	for _, val := range vals {
		if !utf8.ValidString(val) {
			return fmt.Errorf("label value %q is not valid UTF-8", val)
		}
	}

	return nil
}

As far as I know the version of the Prometheus library we use uses UTF-8 validation by default, not legacy.

[electron0zero]

Checking if the label is Valid UTF-8 string will not catch it things like __name__ and labels that start with __, we need to rejected those as well.

If you replace this metric creation code with just utf-8 validation, and you will see that we will allow bunch of invalid label names.

one easy way to see it is to comment out the NewConstMetric the code and add the UTF-8 Validation, and run the TestValidateCostAttributionDimensions unit test.

Creating the metrics allows us to reusing the validations that are sprinkled across client_golang code without re-implementing them.

we will need to also do the validations that are done in https://github.com/prometheus/client_golang/blob/c316de06dea0caf0631f9eca944bc74cca1eb19c/prometheus/desc.go#L92 if want to avoid using NewConstMetric

NewConstMetric should be fine since we only do call this code path on overrides config reload or when we get call the overrides API and modify the cost attribution configs. because of that, I am not too worried about performance of this code.

also, NewConstMetric just creates an ad-hoc metric so it's not registered or exposed anywhere, and won't be collected as well.

[joe-elliott]

I'm good on this PR, but it looks like @carles-grafana has some open questions. Feel free to approve/merge once you two are satisfied.

[carles-grafana]

My point is that creating the metric here is unnecessary, because looking at the function definition the only relevant validation is UTF-8:

func NewConstMetric(desc *Desc, valueType ValueType, value float64, labelValues ...string) (Metric, error) {
	if desc.err != nil {
		return nil, desc.err
	}
	if err := validateLabelValues(labelValues, len(desc.variableLabels.names)); err != nil {
		return nil, err
	}

	metric := &dto.Metric{}
	if err := populateMetric(valueType, value, MakeLabelPairs(desc, labelValues), nil, metric, nil); err != nil {
		return nil, err
	}

	return &constMetric{
		desc:   desc,
		metric: metric,
	}, nil
}

We only care about validating the label name, and we can see that the only relevant validation is the UTF-8, so we just need to check that and we avoid doing extra work:

func validateLabelValues(vals []string, expectedNumberOfValues int) error {
	if len(vals) != expectedNumberOfValues {
		// The call below makes vals escape, copy them to avoid that.
		vals := append([]string(nil), vals...)
		return fmt.Errorf(
			"%w: expected %d label values but got %d in %#v",
			errInconsistentCardinality, expectedNumberOfValues,
			len(vals), vals,
		)
	}

	for _, val := range vals {
		if !utf8.ValidString(val) {
			return fmt.Errorf("label value %q is not valid UTF-8", val)
		}
	}

	return nil
}

As far as I know the version of the Prometheus library we use uses UTF-8 validation by default, not legacy.