-
Notifications
You must be signed in to change notification settings - Fork 658
Open
Labels
bugSomething isn't workingSomething isn't working
Description
What is the bug?
We use these kubernetes-mixin recording rules in Mimir and some of these recording rules get the health error err-mimir-sample-duplicate-timestamp. Full error shown in Grafana is like this
rpc error: code = InvalidArgument desc = send data to ingesters: failed pushing to ingester sx-mimir-ingester-zone-c-0: user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2025-09-23T07:52:37.489Z and is from series apiserver_request:burnrate1d{cluster="democluster"} (sampled 1/10)
currently these rules have errors (the errors disappear and appear sometimes):
in another cluster only these two rules have this errors:
How to reproduce it?
I have this setup, but I guess the problem is easier reproduceable:
- Install mimir, k8s-monitoring and grafana in Kubernetes.
- deploy recording rules https://github.com/suxess-it/kubriX/blob/main/platform-apps/charts/mimir/recording-rules/kubernetes-recording-rules.yaml in mimir via k8s-monitoring with this values.
What did you think would happen?
I expect that the recording rules do not have errors.
What was your environment?
Kubernetes
Mimir Helm Chart (distributed): 5.7.0
Grafana Helm Chart 9.2.10
K8s-monitoring Helm Chart 3.1.5
Any additional context to share?
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working