Avoid large blocks on forced compaction of blocks ingester

[codesome]

What this PR does:

Creates smaller blocks out of Head during idle and forced compaction.

Which issue(s) this PR fixes:
Fixes #3308

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

Thanks @codesome ! Good work, LGTM. I left few minor comments, mostly design. I would also be glad if you could add a unit test: we already have TestIngester_flushing and, if I'm not missing anything, could be easy to add another test case where you push samples spanning over different days and make sure it generates 2 output blocks.

[codesome]

PR is ready for another review. I have added a unit test and refactored the code as discussed.

[pstibrany]

LGTM with some nits.

[pstibrany]

Based on offline discussion with @codesome, I've removed compaction mutex, since it's not needed.

Additionally, I've replaced u.db.Compact() call with a for-loop, since during head compaction (in if block), we could receive more data spanning 2 blocks again, which would again create bigger block in last head compact call. Using for-loop fixes that.

[codesome]

> Using for-loop fixes that.

Actually it does not, it's similar to calling a Compact(). Once you exit the loop here, you can still get more samples after that.

Additionally, a brute force loop might end up creating unaligned block since it is not aligning them like TSDB does (this brings one more thing that we will need to duplicate other checks that TSDB does and keep them in sync, maintenance burden?).

I propose keeping the Compact() call and instead rely on this for no samples during force flushing.

[codesome]

LGTM! This is more robust against what Peter mentioned above. Ignore my previous comment.

[pracucci]

Good job! LGTM, thanks! I just left a couple of minor nits.

[pracucci]

spanning over 3 days

Aren't they over 2 days? However I believe the test would be even stronger if samples would span 3 days and you would expect 3 blocks in output.

[codesome]

However I believe the test would be even stronger if samples would span 3 days and you would expect 3 blocks in output.

It will depend on the actual samples, because by default it will try to create 2h blocks and not over a day. So a test of creating smaller blocks + breaking the block when it spans 2 day might be fine. I will update the test and maybe have 3 days itself anyway.