distributor: log truncated oversized attributes with rate limited logger

[carles-grafana]

What this PR does:

global rate-limited logger for truncation warnings

The truncation log line is a supplementary diagnostic: it tells operators which attribute was truncated, at what scope, and how large it was. The primary observability channel is metricAttributesTruncated, which fires unconditionally per-tenant on every truncation.

This approach guarantees the first truncation event in any time window is always logged.

Rate-limited logger over sampling because at peak load (e.g. 20k push req/s per tenant), sampling at 0.1% still produces 20 log lines/s per pod. The rate-limited logger caps it at 1/s regardless of load.

The log lines are most useful for the noisiest tenant: the one generating the most truncations is the one operators most need diagnostic detail for. If tenant A dominates at 100k truncations/s, that's the tenant we want to see in logs. A quieter tenant B (10/s) is suppressed from logs but fully visible via per-tenant metrics.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[javiermolinar]

I think we are in the good direction bu logging each truncated attribute in the inner loop is too costly for the distributor hot path.

My proposal is sampling plus a single log statement:

• On the first truncation observed, decide sampling rate of 0.001 if you want to log or not the truncated attributes
• Keep truncating and counting regardless of sampling outcome
• Emit one structured log
• Log the result in PushTraces

[carles-grafana]

benchmark comparison with count 8, no significant difference

goos: darwin
goarch: arm64
pkg: github.com/grafana/tempo/modules/distributor
cpu: Apple M3 Pro
                                            │             │
                                            │   sec/op    │
ProcessAttributes/truncation_no_detail-11     54.61µ ± 2%
ProcessAttributes/truncation_with_detail-11   52.16µ ± 2%
geomean                                       53.37µ

                                            │              │
                                            │     B/op     │
ProcessAttributes/truncation_no_detail-11     530.5Ki ± 0%
ProcessAttributes/truncation_with_detail-11   530.5Ki ± 0%
geomean                                       530.5Ki

                                            │            │
                                            │ allocs/op  │
ProcessAttributes/truncation_no_detail-11     400.0 ± 0%
ProcessAttributes/truncation_with_detail-11   400.0 ± 0%
geomean                                       400.0

[javiermolinar]

This is not reliable since most of the time of the benchmark is crafting the test data. We should do it just once and reuse it between iterations

[carles-grafana]

It is reliable in relative terms to show no extra allocations and a tiny overhead of nanoseconds. But you are right that ideally it would be done outside, but the problem is that processAttributes truncates the slice values in-place, so we would need to pre-generate millions of slices. Not doable. So I'm going do remove this once, since it's no longer useful and I pasted the benchstat results in the PR.

[javiermolinar]

I'm ok with this. We could also keep a example per scope but we should measure the impact

[javiermolinar]

Since we want to expose this to tenants should we include the insight=true?

[carles-grafana]

We can do this later once we see it's good from our side