Open
Description
Describe the bug
Our loki write logs are littered with this error now because we are not sending any structured metadata through:
level=error ts=2025-04-22T05:12:50.759646919Z caller=push.go:190 org_id=fake traceID=24274f310f6d627f msg="negative structured metadata bytes received" userID=fake retentionHours=8784 isAggregatedMetric=false policyName= size=0
level=error ts=2025-04-22T05:12:51.856799821Z caller=push.go:190 org_id=fake traceID=58c8970101c4f3b1 msg="negative structured metadata bytes received" userID=fake retentionHours=8784 isAggregatedMetric=false policyName= size=0
level=error ts=2025-04-22T05:12:53.132847816Z caller=push.go:190 org_id=fake traceID=5109f365773bb490 msg="negative structured metadata bytes received" userID=fake retentionHours=8784 isAggregatedMetric=false policyName= size=0
This appears to be added with this change https://github.com/grafana/loki/pull/17056/files.
The prometheus counter Add method supports 0 values, but not negative. We should update this expression to >= so we are not generating unneccessary errors for logs with no structure metadata.
To Reproduce
Steps to reproduce the behavior:
- Run latest weekly version: grafana/loki:k251-87922da
- Send logs with no structured metadata.
Expected behavior
We should update this expression to >= so we are not generating unneccessary errors for requests with no structured metadata.
Environment:
- Infrastructure: kubernetes
- Deployment tool: helm
Screenshots, Promtail config, or terminal output
Logs:
level=error ts=2025-04-22T05:12:50.759646919Z caller=push.go:190 org_id=fake traceID=24274f310f6d627f msg="negative structured metadata bytes received" userID=fake retentionHours=8784 isAggregatedMetric=false policyName= size=0
level=error ts=2025-04-22T05:12:51.856799821Z caller=push.go:190 org_id=fake traceID=58c8970101c4f3b1 msg="negative structured metadata bytes received" userID=fake retentionHours=8784 isAggregatedMetric=false policyName= size=0
level=error ts=2025-04-22T05:12:53.132847816Z caller=push.go:190 org_id=fake traceID=5109f365773bb490 msg="negative structured metadata bytes received" userID=fake retentionHours=8784 isAggregatedMetric=false policyName= size=0