Skip to content

Bug: err-mimir-sample-duplicate-timestamp with some kubernetes-mixin recording rules #12786

@jkleinlercher

Description

@jkleinlercher

What is the bug?

We use these kubernetes-mixin recording rules in Mimir and some of these recording rules get the health error err-mimir-sample-duplicate-timestamp. Full error shown in Grafana is like this

rpc error: code = InvalidArgument desc = send data to ingesters: failed pushing to ingester sx-mimir-ingester-zone-c-0: user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2025-09-23T07:52:37.489Z and is from series apiserver_request:burnrate1d{cluster="democluster"} (sampled 1/10)

currently these rules have errors (the errors disappear and appear sometimes):

Image

in another cluster only these two rules have this errors:

Image

How to reproduce it?

I have this setup, but I guess the problem is easier reproduceable:

What did you think would happen?

I expect that the recording rules do not have errors.

What was your environment?

Kubernetes
Mimir Helm Chart (distributed): 5.7.0
Grafana Helm Chart 9.2.10
K8s-monitoring Helm Chart 3.1.5

Any additional context to share?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions