Batch series in streaming ingester based on message sizes.

[pstibrany]

What this PR does: This PR modifies batching in blocks-based ingester, when it is streaming results back. Instead of using only number of series for batching, it also considers how much data is buffered in memory, and flushes batch if it get too big. This is to prevent problems with too large gRPC messages. "Too big" is hardcoded to 1 MiB, to comfortably fit into grpc default, and also to avoid using too much memory.

Note that if single series returns too much data, it can still reach gRPC limit for sending messages. Solution to that would be splitting single timeseries into multiple messages, but querier doesn't support that at the moment.

Which issue(s) this PR fixes:
Fixes #2945

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

Nice improvement, LGTM! Should we do it for the chunks storage too, to keep it specular? :)

[pstibrany]

Here is a result of benchmark before and after this change.

name                      old time/op    new time/op    deltanchstat before.txt after.txt 
Ingester_v2QueryStream-4    6.31ms ± 2%    6.46ms ± 1%  +2.38%  (p=0.001 n=10+8)

name                      old alloc/op   new alloc/op   delta
Ingester_v2QueryStream-4    3.45MB ± 0%    3.45MB ± 0%  +0.00%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
Ingester_v2QueryStream-4     4.45k ± 0%     4.45k ± 0%  +0.02%  (p=0.000 n=10+10)

[pracucci]

Thanks @pstibrany! LGTM

Should we do it for the chunks storage too, to keep it specular? :)

Could you open an issue for that, please?

[pstibrany]

Could you open an issue for that, please?

I will take a look first if we can make it part of this PR (if it's small change).