Open
Description
Describe the bug
I have the following ruler configuration
ruler:
remote_write:
clients:
victoriametrics:
authorization:
credentials: <random string>
type: Bearer
url: https://vminsert.example.com/prometheus/api/v1/write
enabled: true
But VictoriaMetrics is rejecting this request with HTTP 401 Unauthorized. When investigating the incoming HTTP request via Wireshark, the token is not included, but instead <secret> is used.
This seems to be similar to this issue, but with Bearer token now: #5140
To Reproduce
Steps to reproduce the behavior:
- Started Loki 3.4.2
- Started Prometheus/VictoriaMetrics, where write endpoint is protected with bearer token
- Configured Loki ruler to write recording rules results to Prometheus/VictoriaMetrics using bearer token
- Loki is outputting error
ts=2025-04-07T13:21:32.770110871Z caller=dedupe.go:112 storage=registry manager=tenant-wal instance=fake component=remote level=error remote_name=fake-rw-victoriametrics url=https://vminsert.example.com/prometheus/api/v1/write msg="non-recoverable error" failedSampleCount=63 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 401 Unauthorized: Unauthorized\n"
Expected behavior
Ruler calculated metrics are sent to Prometheus/VictoriaMetrics successfully.
Environment:
- Infrastructure: Kubernetes
- Deployment tool: Helm
/ $ /usr/bin/loki -version
loki, version 3.4.2 (branch: release-3.4.x, revision: 4fa045d3)
build user: root@buildkitsandbox
build date: 2025-02-14T08:42:52Z
go version: go1.23.6
platform: linux/arm64
tags: netgo
Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.