Skip to content

fix: sanitize User-Agent to valid UTF-8 before using as Prometheus label value - #21162

Merged
Segflow merged 1 commit into
mainfrom
meher/distributor-panic-invalid-utf8
Mar 13, 2026
Merged

fix: sanitize User-Agent to valid UTF-8 before using as Prometheus label value#21162
Segflow merged 1 commit into
mainfrom
meher/distributor-panic-invalid-utf8

Conversation

@Segflow

@Segflow Segflow commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Sanitizes the User-Agent HTTP header to valid UTF-8 before passing it to prometheus.CounterVec.WithLabelValues() in the push path. Invalid UTF-8 bytes in the User-Agent (e.g. \xe5\xe5\xe5...) cause WithLabelValues to panic, crashing the distributor.

The fix uses strings.ToValidUTF8(userAgent, "") to strip invalid UTF-8 sequences before the value is used as a Prometheus label.

Which issue(s) this PR fixes:
Fixes distributor panic on push requests with invalid UTF-8 User-Agent headers.

Special notes for your reviewer:
The panic was observed in production with a User-Agent containing raw \xe5 bytes. The regression test uses a recent timestamp to ensure the distributorLagByUserAgent code path is exercised.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory.
@Segflow
Segflow requested a review from a team as a code owner March 13, 2026 14:34
@Segflow
Segflow merged commit ed3cf08 into main Mar 13, 2026
88 checks passed
@Segflow
Segflow deleted the meher/distributor-panic-invalid-utf8 branch March 13, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 participants