Skip to content

vParquet5: virtual span row numbers - #5943

Merged
stoewer merged 18 commits into
grafana:mainfrom
stoewer:vp5-virtual-span-row-numbers
Nov 27, 2025
Merged

vParquet5: virtual span row numbers#5943
stoewer merged 18 commits into
grafana:mainfrom
stoewer:vp5-virtual-span-row-numbers

Conversation

@stoewer

@stoewer stoewer commented Nov 14, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this PR does:
Calculate virtual span row numbers for queries containing only resource attributes

Which issue(s) this PR fixes:
Fixes #5811

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
@stoewer

stoewer commented Nov 14, 2025

Copy link
Copy Markdown
Contributor Author

Benchmark results:

goos: linux
goarch: amd64
pkg: github.com/grafana/tempo/tempodb/encoding/vparquet5
cpu: AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M       
                                                   │ bench-virtrows-01-vp5.txt │   bench-virtrows-01-vp5-virt.txt   │
                                                   │          sec/op           │   sec/op     vs base               │
BackendBlockTraceQL/spanAttValMatch-12                             255.9m ± 4%   256.6m ± 2%        ~ (p=0.574 n=8)
BackendBlockTraceQL/spanAttValMatchFew-12                          118.7m ± 2%   118.2m ± 1%        ~ (p=0.798 n=8)
BackendBlockTraceQL/unscopedAttValMatch-12                         954.2m ± 2%   870.3m ± 2%   -8.79% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValMatch-12                         543.9m ± 1%   503.3m ± 2%   -7.47% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValNoMatch-12                       10.89m ± 3%   11.19m ± 2%   +2.75% (p=0.003 n=8)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12                   93.43m ± 2%   89.46m ± 2%   -4.25% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12                 6.206m ± 6%   6.094m ± 5%        ~ (p=0.505 n=8)
BackendBlockTraceQL/mixedValNoMatch-12                             545.1m ± 2%   462.0m ± 3%  -15.24% (p=0.000 n=8)
BackendBlockTraceQL/mixedValMixedMatchAnd-12                       6.730m ± 4%   6.633m ± 4%        ~ (p=0.798 n=8)
BackendBlockTraceQL/mixedValMixedMatchOr-12                        527.5m ± 2%   432.0m ± 2%  -18.10% (p=0.000 n=8)
BackendBlockTraceQL/count-12                                        1.737 ± 1%    1.666 ± 2%   -4.08% (p=0.000 n=8)
BackendBlockTraceQL/struct-12                                       1.728 ± 2%    1.640 ± 2%   -5.12% (p=0.000 n=8)
BackendBlockTraceQL/||-12                                          848.5m ± 2%   769.7m ± 1%   -9.28% (p=0.000 n=8)
BackendBlockTraceQL/mixed-12                                       217.0m ± 2%   216.0m ± 2%        ~ (p=0.442 n=8)
BackendBlockTraceQL/complex-12                                     430.0m ± 1%   430.2m ± 2%        ~ (p=0.878 n=8)
BackendBlockTraceQL/select-12                                       1.162 ± 2%    1.117 ± 3%   -3.89% (p=0.000 n=8)
geomean                                                            228.0m        216.6m        -5.00%

                                                   │ bench-virtrows-01-vp5.txt │   bench-virtrows-01-vp5-virt.txt    │
                                                   │            B/s            │     B/s       vs base               │
BackendBlockTraceQL/spanAttValMatch-12                            390.1Mi ± 4%   389.2Mi ± 4%        ~ (p=0.574 n=8)
BackendBlockTraceQL/spanAttValMatchFew-12                         841.0Mi ± 2%   844.3Mi ± 2%        ~ (p=0.798 n=8)
BackendBlockTraceQL/unscopedAttValMatch-12                        112.5Mi ± 2%   122.7Mi ± 2%   +9.04% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValMatch-12                        184.5Mi ± 2%   198.3Mi ± 2%   +7.44% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValNoMatch-12                      794.1Mi ± 2%   772.9Mi ± 2%   -2.67% (p=0.003 n=8)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12                  563.4Mi ± 3%   585.3Mi ± 2%   +3.89% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12                282.1Mi ± 3%   287.3Mi ± 4%        ~ (p=0.505 n=8)
BackendBlockTraceQL/mixedValNoMatch-12                            26.04Mi ± 2%   29.44Mi ± 4%  +13.04% (p=0.000 n=8)
BackendBlockTraceQL/mixedValMixedMatchAnd-12                      1.026Gi ± 5%   1.041Gi ± 5%        ~ (p=0.798 n=8)
BackendBlockTraceQL/mixedValMixedMatchOr-12                       21.72Mi ± 3%   25.14Mi ± 2%  +15.77% (p=0.000 n=8)
BackendBlockTraceQL/count-12                                      57.72Mi ± 1%   59.82Mi ± 2%   +3.63% (p=0.000 n=8)
BackendBlockTraceQL/struct-12                                     66.75Mi ± 2%   69.98Mi ± 2%   +4.84% (p=0.000 n=8)
BackendBlockTraceQL/||-12                                         119.0Mi ± 1%   130.4Mi ± 1%   +9.59% (p=0.000 n=8)
BackendBlockTraceQL/mixed-12                                      535.9Mi ± 2%   538.4Mi ± 3%        ~ (p=0.442 n=8)
BackendBlockTraceQL/complex-12                                    468.5Mi ± 2%   468.4Mi ± 2%        ~ (p=0.903 n=8)
BackendBlockTraceQL/select-12                                     72.53Mi ± 3%   74.94Mi ± 2%   +3.32% (p=0.000 n=8)
geomean                                                           194.7Mi        203.2Mi        +4.37%

                                                   │ bench-virtrows-01-vp5.txt │   bench-virtrows-01-vp5-virt.txt   │
                                                   │         MB_io/op          │  MB_io/op   vs base                │
BackendBlockTraceQL/spanAttValMatch-12                              104.7 ± 0%   104.7 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/spanAttValMatchFew-12                           104.7 ± 0%   104.7 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/unscopedAttValMatch-12                          112.6 ± 0%   112.0 ± 0%  -0.53% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValMatch-12                          105.2 ± 0%   104.6 ± 0%  -0.57% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValNoMatch-12                        9.067 ± 0%   9.067 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/resourceAttIntrinsicMatch-12                    55.18 ± 0%   54.90 ± 0%  -0.51% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12                  1.835 ± 0%   1.835 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/mixedValNoMatch-12                              14.89 ± 0%   14.26 ± 0%  -4.23% (p=0.000 n=8)
BackendBlockTraceQL/mixedValMixedMatchAnd-12                        7.414 ± 0%   7.414 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/mixedValMixedMatchOr-12                         12.01 ± 0%   11.39 ± 0%  -5.16% (p=0.000 n=8)
BackendBlockTraceQL/count-12                                        105.1 ± 0%   104.5 ± 0%  -0.57% (p=0.000 n=8)
BackendBlockTraceQL/struct-12                                       120.9 ± 0%   120.3 ± 0%  -0.50% (p=0.000 n=8)
BackendBlockTraceQL/||-12                                           105.9 ± 0%   105.3 ± 0%  -0.57% (p=0.000 n=8)
BackendBlockTraceQL/mixed-12                                        122.0 ± 0%   122.0 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/complex-12                                      211.3 ± 0%   211.3 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/select-12                                       88.36 ± 0%   87.73 ± 0%  -0.71% (p=0.000 n=8)
geomean                                                             46.54        46.15       -0.85%
¹ all samples are equal

                                                   │ bench-virtrows-01-vp5.txt │   bench-virtrows-01-vp5-virt.txt    │
                                                   │         spans/op          │  spans/op    vs base                │
BackendBlockTraceQL/spanAttValMatch-12                           18.28k ± 0%     18.28k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/spanAttValMatchFew-12                        2.364k ± 0%     2.364k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/unscopedAttValMatch-12                       49.93k ± 0%     49.93k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/resourceAttValMatch-12                       49.93k ± 0%     53.77k ± 0%  +7.69% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValNoMatch-12                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/resourceAttIntrinsicMatch-12                 4.586k ± 0%     4.587k ± 0%  +0.02% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12                0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/mixedValNoMatch-12                            0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/mixedValMixedMatchAnd-12                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/mixedValMixedMatchOr-12                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/count-12                                     231.9k ± 0%     231.9k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/struct-12                                     84.00 ± 0%      84.00 ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/||-12                                        11.64k ± 0%     11.64k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/mixed-12                                     1.910k ± 0%     1.910k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/complex-12                                   3.571k ± 0%     3.571k ± 0%       ~ (p=1.000 n=8) ¹
BackendBlockTraceQL/select-12                                    1.184k ± 0%     1.189k ± 0%  +0.42% (p=0.000 n=8)
geomean                                                                      ²                +0.49%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                   │ bench-virtrows-01-vp5.txt │    bench-virtrows-01-vp5-virt.txt    │
                                                   │           B/op            │     B/op       vs base               │
BackendBlockTraceQL/spanAttValMatch-12                           103.6Mi ±  2%   103.4Mi ±  1%        ~ (p=0.721 n=8)
BackendBlockTraceQL/spanAttValMatchFew-12                        17.47Mi ±  2%   18.44Mi ±  6%   +5.55% (p=0.002 n=8)
BackendBlockTraceQL/unscopedAttValMatch-12                       328.9Mi ±  1%   327.3Mi ±  2%        ~ (p=0.195 n=8)
BackendBlockTraceQL/resourceAttValMatch-12                       322.3Mi ±  1%   325.1Mi ±  0%   +0.86% (p=0.005 n=8)
BackendBlockTraceQL/resourceAttValNoMatch-12                     12.72Mi ±  1%   12.73Mi ±  1%        ~ (p=0.505 n=8)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12                 22.00Mi ±  1%   22.31Mi ±  2%        ~ (p=0.083 n=8)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12               7.044Mi ±  0%   7.002Mi ±  1%   -0.60% (p=0.050 n=8)
BackendBlockTraceQL/mixedValNoMatch-12                           19.81Mi ± 27%   13.52Mi ± 40%  -31.77% (p=0.001 n=8)
BackendBlockTraceQL/mixedValMixedMatchAnd-12                     7.523Mi ±  1%   7.508Mi ±  1%        ~ (p=0.442 n=8)
BackendBlockTraceQL/mixedValMixedMatchOr-12                      14.48Mi ± 11%   12.93Mi ± 48%        ~ (p=0.442 n=8)
BackendBlockTraceQL/count-12                                     2.069Gi ±  1%   2.057Gi ±  1%   -0.56% (p=0.003 n=8)
BackendBlockTraceQL/struct-12                                    191.1Mi ±  8%   193.0Mi ±  2%        ~ (p=0.442 n=8)
BackendBlockTraceQL/||-12                                        499.9Mi ±  1%   496.5Mi ±  2%        ~ (p=0.234 n=8)
BackendBlockTraceQL/mixed-12                                     24.62Mi ±  6%   24.47Mi ±  6%        ~ (p=0.645 n=8)
BackendBlockTraceQL/complex-12                                   58.96Mi ±  7%   58.98Mi ±  2%        ~ (p=0.878 n=8)
BackendBlockTraceQL/select-12                                    1.464Gi ±  0%   1.454Gi ±  1%   -0.70% (p=0.000 n=8)
geomean                                                          70.34Mi         68.40Mi         -2.76%

                                                   │ bench-virtrows-01-vp5.txt │  bench-virtrows-01-vp5-virt.txt   │
                                                   │         allocs/op         │  allocs/op   vs base              │
BackendBlockTraceQL/spanAttValMatch-12                             898.9k ± 0%   898.9k ± 0%       ~ (p=0.425 n=8)
BackendBlockTraceQL/spanAttValMatchFew-12                          159.4k ± 0%   159.4k ± 0%  +0.01% (p=0.001 n=8)
BackendBlockTraceQL/unscopedAttValMatch-12                         2.625M ± 0%   2.624M ± 0%  -0.06% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValMatch-12                         2.628M ± 0%   2.624M ± 0%  -0.14% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttValNoMatch-12                       70.61k ± 0%   70.61k ± 0%  +0.00% (p=0.008 n=8)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12                   256.8k ± 0%   244.4k ± 0%  -4.83% (p=0.000 n=8)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12                 70.54k ± 0%   70.54k ± 0%       ~ (p=1.000 n=8)
BackendBlockTraceQL/mixedValNoMatch-12                             73.11k ± 0%   71.36k ± 0%  -2.39% (p=0.009 n=8)
BackendBlockTraceQL/mixedValMixedMatchAnd-12                       70.78k ± 0%   70.78k ± 0%       ~ (p=0.327 n=8)
BackendBlockTraceQL/mixedValMixedMatchOr-12                        73.06k ± 0%   71.36k ± 0%  -2.32% (p=0.010 n=8)
BackendBlockTraceQL/count-12                                       13.14M ± 0%   13.14M ± 0%  -0.02% (p=0.000 n=8)
BackendBlockTraceQL/struct-12                                      1.120M ± 0%   1.118M ± 0%  -0.15% (p=0.050 n=8)
BackendBlockTraceQL/||-12                                          2.760M ± 0%   2.758M ± 0%  -0.06% (p=0.010 n=8)
BackendBlockTraceQL/mixed-12                                       201.8k ± 0%   201.8k ± 0%       ~ (p=0.442 n=8)
BackendBlockTraceQL/complex-12                                     398.0k ± 0%   398.0k ± 0%       ~ (p=0.169 n=8)
BackendBlockTraceQL/select-12                                      6.570M ± 0%   6.570M ± 0%  -0.01% (p=0.000 n=8)
geomean
@stoewer
stoewer marked this pull request as ready for review November 14, 2025 02:58
joe-elliott
joe-elliott reviewed Nov 14, 2025
View reviewed changes
Comment thread tempodb/encoding/vparquet5/block_traceql.go
Comment thread tempodb/encoding/vparquet5/row_number_iterator.go
Comment thread tempodb/encoding/vparquet5/row_number_iterator.go Outdated
joe-elliott
joe-elliott reviewed Nov 14, 2025
View reviewed changes
Comment thread tempodb/encoding/vparquet5/row_number_iterator.go
joe-elliott
joe-elliott approved these changes Nov 26, 2025
View reviewed changes

@joe-elliott joe-elliott left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spoke offline and understand the choices made in the virtual row number iterator. Seems like a few bugs were fixed and we're ready to merge!
@stoewer
stoewer merged commit 7b00d6c into grafana:main Nov 27, 2025
21 checks passed
@knylander-grafana knylander-grafana mentioned this pull request Jan 9, 2026
Merged
3 tasks
@stoewer stoewer mentioned this pull request Feb 10, 2026
Merged
3 tasks
@stoewer
stoewer deleted the vp5-virtual-span-row-numbers branch February 12, 2026 09:52
mattdurham pushed a commit to mattdurham/tempo that referenced this pull request Jun 18, 2026
@stoewer
vParquet5: virtual span row numbers (grafana#5943)
b868712 
* Add field SpanCount to ScopeSpans

* Make new vp5 preview version

* Convert vp5 test-data

* Move rowNumberIterator to own file

* Implement virtualRowNumberIterator

* Improve tests for virtualRowNumberIterator

* Set correct span count for vp5 test data

* Use virtualRowNumberIterator instead of StatusCode colum iter

* CHANGELOG.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants

@stoewer @joe-elliott @mdisibio