[Feature] traceql comparison operators by ruslan-mikhailov · Pull Request #6474 · grafana/tempo

ruslan-mikhailov · 2026-02-16T17:14:59Z

What this PR does: it introduces comparison queries to TraceQL Metrics such as {} | rate() > 10. It is sensitive to step variable, especially to queries like count_over_time().
Examples:
No filters

More than 1

More than 3. In this example you can see that Grafana connects dots even when NaN is between them. This should be addressed in a separate issue (proposal is here #6486).

How it works if Tempo returns NaN (not included in this PR!)

Duration

Combination with topk

Which issue(s) this PR fixes:
Fixes # #6417

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

ruslan-mikhailov · 2026-02-17T14:28:49Z

+// Uses == for equality instead of = to distinguish from field-level equality
+// **********************
+metricsFilterOperation:
+    EQ         { $$ = OpEqual        }


I picked = and not == to align with how we compare attributes

I think starting with consistency is good. If for some reason it's not fitting well, we could change it or support both. Fwiw this operator is likely to not be used much.

ruslan-mikhailov · 2026-02-17T14:29:34Z

 %type <scalarFilterOperation> scalarFilterOperation
 %type <metricsAggregation> metricsAggregation
 %type <metricsSecondStage> metricsSecondStage
+%type <scalarFilterOperation> metricsFilterOperation


After we agree on syntax, we can add the feature to documentation

javiermolinar

Nice feature! I missed some benchmarks

javiermolinar · 2026-02-17T15:13:19Z

 		},
+		// Combined second stage tests (topk/bottomk with filter)
+		{
+			in: `{ } | rate() | topk(5) > 10`,


Can we invert the order?

{ } | rate() > 10 | topk(5)

In the current solution, it is not possible. I can implement, but do we actually need this feature? 🤔

I guess so? I think allowing both is the best user experience. Right now it will be confussing why we allow one form and not the other

Yes agree, I would expect to be able to use the scalar filter on any part of the pipeline which is metrics. Also the queries { } | rate() | topk(5) > 10 and { } | rate() > 10 | topk(5) are meaningfully different.

I think we could add the scalar filter as another second stage function, so that { } | rate() > 10 | topk(5) < 11 is parsed as:

spanpipeine: { }
first stage: rate()
second stages: > 10, topk(5), < 11

An argument could be made to add the scalar filter inside a stage. Depending on if that has very large implementation benefits.

If this helps, here is a query with multiple metrics stages and regrouping that I would like to support eventually:

{ } | rate() by (pod) > 10 | topk(5) by (namespace) | sum() by (cluster) > 20

I concur with the above. This feature would benefit from more time on drawing board to ensure more cases are supported, or there is a concrete plan on how to achieve that

Added support for { } | rate() > 10 | topk(5) and minor refactoring for better support

This is so close to being generalized for N stages:

// ChainedSecondStage chains two second stage elements together. // The first element is processed, then its output is passed to the second. // Example: {status=error} | rate() | topk(5) > 10 type ChainedSecondStage struct { first secondStageElement second secondStageElement }

Encountered this somewhat quickly by trying to do a query with rate() > X | topk(5) < Y. I do think the PR as-is could be confusing to users, and it is worth generalizing. For example it is not clear why a comparison operator works with topk in the first query but not the second, without understanding the current parsing:

rate() | topk(5) < 100 rate() > 0 | topk(5) < 100

Oh, I misunderstood you. Added, now it's even more clean

javiermolinar · 2026-02-17T15:51:00Z

+
+		exemplars := make([]Exemplar, 0, len(series.Exemplars))
+		for i, e := range series.Exemplars {
+			if m.compare(e.Value) {


We need to guard here against NaN too since NaN != 10 is true

Nice, catch! Fixed by dropping such exemplars

ruslan-mikhailov · 2026-02-18T12:20:44Z

+ review fixes
+ tests for .String()

ruslan-mikhailov · 2026-02-23T10:42:07Z

+ review fixes

knylander-grafana · 2026-02-26T16:17:57Z

Do we need docs for this?

mdisibio

This is looking great. 95% lgtm, just want to see if we can update a few small places.

mdisibio · 2026-03-02T18:02:08Z

+
+	for key, series := range input {
+		hasValue := false
+		newValues := make([]float64, len(series.Values))


For performance, I think it would be ok to edit the input series here. Is there any case where the same series is reused elsewhere?

I think they should not? Calling Result twice with MetricsFilter will also have the same result due to IsNaN check.

Pushed in a separate commit

mdisibio · 2026-03-02T18:05:38Z

+}
+
+func (c ChainedSecondStage) separator() string {
+	return c[0].separator()


This isn't used, which makes me consider if separator() shouldn't be part of the interface of individual elements. But part of the second stage, to be used in String() function above only.

i.e. instead of appending directly in expr.y with append($1, $3), we could add a helper that appends with a given separator.

mdisibio · 2026-03-02T18:07:25Z

+  | spansetPipelineExpression                                             { yylex.(*lexer).expr = newRootExpr($1) }
+  | scalarPipelineExpressionFilter                                        { yylex.(*lexer).expr = newRootExpr($1) } 
+  | spansetPipeline PIPE metricsAggregation                               { yylex.(*lexer).expr = newRootExprWithMetrics($1, $3) }
+  | spansetPipeline PIPE metricsAggregation metricsSecondStagePipeline    { yylex.(*lexer).expr = newRootExprWithMetricsTwoStage($1, $3, secondStagePipeline($4)) }


This is fantastic 👍 One of these days we could rename metricsAggregation to metricsFirstStagePipeline and I think it would be very easy to understand.

mdisibio

I'm loving this PR. BTW this works too {} | rate() by (resource.service.name) > 20 < 60 🚀

LGTM

ruslan-mikhailov · 2026-03-04T09:08:05Z

No-op push to retrigger CI pipeline

Introduce comparison operators to TraceQL Metrics. Examples: {} | rate() by (resource.service.name) < 0.01 {} | avg_over_time(span:duration) > 1s

ruslan-mikhailov changed the title ~~Feature/traceql comparison~~ Feb 16, 2026

ruslan-mikhailov force-pushed the feature/traceql-comparison branch 5 times, most recently from 68f1bc4 to 21cd34c Compare February 17, 2026 12:39

ruslan-mikhailov commented Feb 17, 2026

View reviewed changes

ruslan-mikhailov marked this pull request as ready for review February 17, 2026 14:33

ruslan-mikhailov requested review from carles-grafana, electron0zero, ie-pham, javiermolinar, mapno, mattdurham, mdisibio, oleg-kozlyuk-grafana, stoewer, yvrhdn and zalegrala as code owners February 17, 2026 14:33

javiermolinar reviewed Feb 17, 2026

View reviewed changes

ruslan-mikhailov mentioned this pull request Feb 17, 2026

Proposal: [Enhancement] TraceQL Metrics: return NaN values #6486

Closed

3 tasks

ruslan-mikhailov added 2 commits February 18, 2026 13:17

Support comparison operators in AST

aebd9a6

make gen-traceql

a3b645e

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from 06a732d to a5ebfea Compare February 18, 2026 12:19

ruslan-mikhailov requested a review from javiermolinar February 18, 2026 12:20

electron0zero reviewed Feb 19, 2026

View reviewed changes

ruslan-mikhailov added 3 commits February 23, 2026 11:38

Implement metrics comparison filter

6fb83e2

Unit tests

89655b9

Support comparison with duration

5fb4c61

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from a5ebfea to bd1f47c Compare February 23, 2026 10:41

ruslan-mikhailov added 2 commits February 23, 2026 13:46

Support comparison -> topk/bottomk

0ab3188

make gen-traceql

9ad86f1

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from ca50186 to 75aea14 Compare February 23, 2026 13:07

ruslan-mikhailov added 2 commits February 23, 2026 15:55

Unit tests for comparison -> topk

a6bfa42

Better String for second pass

17b15ff

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from 75aea14 to 17b15ff Compare February 23, 2026 14:56

ruslan-mikhailov added 3 commits February 25, 2026 08:55

general chained second stage

151525e

apply general second stage

f89fa7c

make gen-traceql

f5c7fdf

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from 5177cf6 to 858a04a Compare February 25, 2026 08:28

mdisibio reviewed Mar 2, 2026

View reviewed changes

Unit tests

feb90ad

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from 858a04a to 33b0c79 Compare March 3, 2026 09:25

ruslan-mikhailov added 2 commits March 3, 2026 13:24

Optimisation: do not copy values, change in place

960f2ff

refactoring: separator

ce71087

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from 33b0c79 to d0d43a4 Compare March 3, 2026 12:57

mdisibio approved these changes Mar 3, 2026

View reviewed changes

make gen-traceql

b92c0bf

ruslan-mikhailov force-pushed the feature/traceql-comparison branch from d0d43a4 to b92c0bf Compare March 4, 2026 09:07

ruslan-mikhailov enabled auto-merge (squash) March 4, 2026 09:16

ruslan-mikhailov disabled auto-merge March 4, 2026 09:17

ruslan-mikhailov merged commit 3d3f747 into grafana:main Mar 4, 2026
24 checks passed

ruslan-mikhailov mentioned this pull request Mar 12, 2026

Docs: comparison operators #6659

Merged

3 tasks

This was referenced Jun 5, 2026

feat: support syntax for TraceQL Math grafana/lezer-traceql#80

Draft

[Epic] TraceQL Math #6417

Open

Uh oh!

Conversation

ruslan-mikhailov commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mdisibio Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javiermolinar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javiermolinar Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

ruslan-mikhailov commented Feb 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ruslan-mikhailov commented Feb 23, 2026

knylander-grafana commented Feb 26, 2026

mdisibio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

mdisibio left a comment

Choose a reason for hiding this comment

ruslan-mikhailov commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Labels

6 participants

ruslan-mikhailov commented Feb 16, 2026 •

edited

Loading

mdisibio Feb 25, 2026 •

edited

Loading

javiermolinar left a comment •

edited

Loading

javiermolinar Feb 18, 2026 •

edited

Loading

ruslan-mikhailov commented Mar 4, 2026 •

edited

Loading