Describe the bug
We are using lz4 compression for chunks. While querying data we are encountering a memory leak on the queriers which is causing the the OOM killer to kick in at around 30GB. Since switching to snappy compression we are no longer encountering this memory leak.
This may be related to issue #6501.
To Reproduce
Steps to reproduce the behavior:
- Started Loki 2.9.2 (compiled with pprof)
- Execute a query for the past week (processing about 80GB of data)
- Ensure the log volume pane is expanded in Grafana to generate additional load
- Querier pod memory increases until the OOM killer kicks in (at around 30GB)
Expected behavior
We expect the memory usage of the queriers while processing lz4 compressed data to be somewhat similar to that of snappy.
Environment:
- Infrastructure: EKS, S3
- Deployment tool: Custom Kubernetes manifests, micro-services model
- GOMEMLIMIT environment variable set to 6GiB
Screenshots, Promtail config, or terminal output
Example pprof heap output while encountering the issue:
Showing nodes accounting for 23643.30MB, 99.24% of 23824.01MB total
Dropped 234 nodes (cum <= 119.12MB)
Showing top 10 nodes out of 43
flat flat% sum% cum cum%
20951.03MB 87.94% 87.94% 20951.03MB 87.94% github.com/pierrec/lz4/v4/internal/lz4block.glob..func6
2688.07MB 11.28% 99.22% 2696.62MB 11.32% github.com/grafana/loki/pkg/storage/chunk/client.(*client).getChunk
5.70MB 0.016% 99.24% 1106.92MB 4.65% github.com/grafana/loki/pkg/logql.(*batchRangeVectorIterator).load
0.50MB 0.0021% 99.24% 10043.87MB 42.16% github.com/pierrec/lz4/v4/internal/lz4stream.NewFrameDataBlock (inline)
0 0% 99.24% 20952.57MB 87.95% bufio.(*Reader).Read
0 0% 99.24% 20982.79MB 88.07% github.com/grafana/loki/pkg/chunkenc.(*bufferedIterator).Next
0 0% 99.24% 20967.23MB 88.01% github.com/grafana/loki/pkg/chunkenc.(*bufferedIterator).moveNext
0 0% 99.24% 20982.79MB 88.07% github.com/grafana/loki/pkg/chunkenc.(*sampleBufferedIterator).Next
0 0% 99.24% 15759.47MB 66.15% github.com/grafana/loki/pkg/iter.(*cachedSampleIterator).Next
0 0% 99.24% 15759.47MB 66.15% github.com/grafana/loki/pkg/iter.(*cachedSampleIterator).consumeWrapped
Describe the bug
We are using lz4 compression for chunks. While querying data we are encountering a memory leak on the queriers which is causing the the OOM killer to kick in at around 30GB. Since switching to snappy compression we are no longer encountering this memory leak.
This may be related to issue #6501.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
We expect the memory usage of the queriers while processing lz4 compressed data to be somewhat similar to that of snappy.
Environment:
Screenshots, Promtail config, or terminal output
Example pprof heap output while encountering the issue: