Rewrite conditions on same attribute to equivalent array operation (2) - #6353
Merged
Conversation
Example ["aa", "bb"] -> [`aa`, `bb`]
The rewriter handles the following cases:
- { .a = "a" || .a = "b" } => { .a IN ["a", "b"] }
- { .a != "a" && .a != "b" } => { .a NOT IN ["a", "b"] }
- { .a =~ "a" || .a =~ "b" } => { .a MATCH ANY ["a", "b"] }
- { .a !~ "a" && .a !~ "b" } => { .a MATCH NONE ["a", "b"] }
BinaryOperation.validate() considered everythin a valid regex where FieldExpression.EncodeToString(false) happened to return a valid regex. With this change the RHS of a regex operation is required to be a string or string array
Contributor
Author
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
stoewer
marked this pull request as ready for review
January 30, 2026 13:33
stoewer
requested review from
electron0zero,
joe-elliott,
mapno,
mdisibio,
yvrhdn and
zalegrala
as code owners
January 30, 2026 13:33
mdisibio
reviewed
Feb 4, 2026
mdisibio
reviewed
Feb 4, 2026
mdisibio
reviewed
Feb 4, 2026
Contributor
Author
|
I ran some benchmarks:
The first 4 sub-benchmarks are just simple queries where nothing can be optimized Results: |
Contributor
Author
|
There are two things in the benchmark that I still need to look into:
|
This fixes additional span/op when optimizations are enabled
Contributor
Author
|
Here are the benchmark results after optimization and fixing the span/op bug. There are again 3 different benchmark runs with the following parameters:
The first 4 sub-benchmarks are just simple queries where nothing can be optimized: |
Contributor
Author
|
Commit bc3b95b fixes a bug that was introduced with the virtual row numbers PR #5943. This is what caused the additional |
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
mdisibio
reviewed
Feb 10, 2026
mdisibio
left a comment
mdisibio
left a comment
Contributor
There was a problem hiding this comment.
Lgtm after latest changes, thanks. Discussed offline and we want to dig into the virtual row number fix a little more before merging.
This helps avoid additional spans/op in queries with only resource attributes
Contributor
Author
|
Latest benchmark results with the new fix for the additional spans/op: |
stoewer
added a commit
to stoewer/tempo
that referenced
this pull request
Feb 11, 2026
This helps avoid additional spans/op in queries with only resource attributes This change part of grafana#6353
This was referenced Feb 11, 2026
stoewer
added a commit
to stoewer/tempo
that referenced
this pull request
Feb 11, 2026
This helps avoid additional spans/op in queries with only resource attributes This fix was part of grafana#6353
3 tasks
mdisibio
approved these changes
Feb 11, 2026
stoewer
added a commit
that referenced
this pull request
Feb 11, 2026
* Read to first virtual row after seeking on a low definition level This helps avoid additional spans/op in queries with only resource attributes This fix was part of #6353 * CHANGELOG.md
3 tasks
3 tasks
mattdurham
pushed a commit
to mattdurham/tempo
that referenced
this pull request
Jun 18, 2026
…rafana#6353) * Add operators IN, NOT IN, MATCH ANY, and MATCH NONE * Implement Static.append() * Print string arrays with '`' instead of '"' Example ["aa", "bb"] -> [`aa`, `bb`] * Rewrite binary operations to array equivalent The rewriter handles the following cases: - { .a = "a" || .a = "b" } => { .a IN ["a", "b"] } - { .a != "a" && .a != "b" } => { .a NOT IN ["a", "b"] } - { .a =~ "a" || .a =~ "b" } => { .a MATCH ANY ["a", "b"] } - { .a !~ "a" && .a !~ "b" } => { .a MATCH NONE ["a", "b"] } * Add test cases with arrays to TestBinOp * traceql.Parse() applies AST optimizations by default * Rename predicate_test.go to predicates_test.go * Add array predicates to parquetquery * BREAKING CHANGE: more precise regex validation BinaryOperation.validate() considered everythin a valid regex where FieldExpression.EncodeToString(false) happened to return a valid regex. With this change the RHS of a regex operation is required to be a string or string array * vp4: handle predicates with arrays correctly * vp5: handle predicates with arrays correctly * vp3: handle predicates with arrays correctly * Add test cases for optimized queries * Update query test examples with optimizable queries * vp3: fix handling of array types * CHANGELOG.md * Pass array operators down to fetch layer * Read to first virtual row after seeking on a low definition level This helps avoid additional spans/op in queries with only resource attributes --------- Co-authored-by: Martin Disibio <mdisibio@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Rewrite queries like
{ .a="a" || .a="b" }into the equivalent, faster form{ .a IN ["a", "b"] }. This AST optimization is applied by default, but it can be disabled using the query hintskip_optimization=true.I've added the new operators IN, NOT IN, MATCH ANY, MATCH NONE to the AST. However, they are not actually parseable in TraceQL.
Note: This PR was created from the changes in #5827. Since reviewers requested larger changes and rebasing caused lots of conflicts, I decided to create a new PR instead of updating the new one
Limitations
{ .a="a" && .b="b" || .a="c" }{ .a IN ["a", "b"] }only works internally, the syntax is not actually available in TraceQLBreaking Changes
!=and `!~' slightly:!=now means NOT IN (previously: CONTAINS NOT EQUAL)!~now means MATCH NONE (previously: CONTAINS NON-MATCH)Which issue(s) this PR fixes:
Fixes grafana/tempo-squad#874
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]