Skip to content

perf(approx_topk): Reduce memory usage of HyperLogLog in approx_topk. #15559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jan 9, 2025
Merged

perf(approx_topk): Reduce memory usage of HyperLogLog in approx_topk. #15559

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0937d39
Benchmark full pipeline.
jeschkies Dec 24, 2024
c2c4efa
Define log generator
jeschkies Dec 24, 2024
388664d
Benchmark
jeschkies Dec 24, 2024
1127c3f
Note todo
jeschkies Dec 24, 2024
22835ff
Avoid allocations by using slices.SortFunc
jeschkies Dec 27, 2024
448aafe
Use non sparse HLL to avoid allocations
jeschkies Dec 27, 2024
5039b1d
Reduce HLL memory usage
jeschkies Dec 30, 2024
081ebef
Merge remote-tracking branch 'grafana/main' into karsten/benchmark-en…
jeschkies Dec 30, 2024
be07020
Revert return
jeschkies Dec 30, 2024
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use non sparse HLL to avoid allocations
  • Loading branch information
@jeschkies
jeschkies committed Dec 27, 2024
commit 448aafe83823fdca175e1e817be5b241edc0fea2
3 changes: 1 addition & 2 deletions pkg/logql/count_min_sketch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,9 +241,8 @@ func (v *HeapCountMinSketchVector) Push(x any) {
func (v *HeapCountMinSketchVector) Pop() any {
old := v.Metrics
n := len(old)
x := old[n-1]
v.Metrics = old[0 : n-1]
return x
return old[n-1]
}

// JoinCountMinSketchVector joins the results from stepEvaluator into a CountMinSketchVector.
Expand Down
2 changes: 1 addition & 1 deletion pkg/logql/sketch/cms.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ func NewCountMinSketch(w, d uint32) (*CountMinSketch, error) {
Depth: d,
Width: w,
Counters: make2dslice(w, d),
HyperLogLog: hyperloglog.New16(),
HyperLogLog: hyperloglog.New16NoSparse(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should leave a comment.

// Sparse HLL sketches should result in less memory usage for cardinalities of 100k or less but the automatic transition from sparse
// to non-sparse sketches above that cardinality range results in significantly more memory allocs/bytes. 
// Until we have a reliable way of estimating the cardinality set in advance, always use non-sparse for faster performance.
}, nil
}

Expand Down
Loading