Compact idle blocks.

[pstibrany]

What this PR does: This PR adds compaction of blocks that are idle for longer than specified timeout.

Draft: needs tests.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pracucci]

Good job Peter! You're rocking this work!

Overall LGTM (just left few minor comments).

I think there's one missing thing. This feature is useful when enabling shuffle sharding or a tenant stop writing. The next ingester startup, we don't want to load the TSDB at all if all existing blocks on disk have passed the retention period. This part is missing (perfectly fine doing it in a separate PR, but I think should be done).

[pstibrany]

The next ingester startup, we don't want to load the TSDB at all if all existing blocks on disk have passed the retention period. This part is missing (perfectly fine doing it in a separate PR, but I think should be done).

I agree this part is missing. I would like to address that in separate PR.

In this PR, I don't want to actually close idle TSDB, because that would cause issues with the way how we collect metrics, esp. in case we need to reopen it. But ignoring on startup is a good way to handle it, I think.

[pracucci]

LGTM, thanks!