fix: Mitigate ingester race between Query & GetChunkIDs

[benclive]

What this PR does / why we need it:

Fixes a race condition between the Query and GetChunkIDs calls when using partition-ingesters.

The bug occurs when first Querying logs from an ingester that just flushed a chunk (returning 0 logs), followed by calling GetChunkIDs on an ingester that has not flushed a chunk (returning 0 chunk refs). The bug manifests by returning 0 results when some log lines do exist in other ingesters.

• This solution is to query all the available ingesters for GetChunkIDs in order to make sure we receive any recent chunks that have just been flushed.
• There is still a small window where a shutting down ingester would flush all it's chunk and then be unavailable during restart to serve the GetChunkIDs request. This is the same with the classic ingesters and would need some more work to fully mitigate, so I propose this as an interim solution while a more complete solution can be discussed & implemented.

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[grobinson-grafana]

I think redundant comment. If we do want a comment I would write something like this (finishing the sentence):

Suggested change

	// Nothing here
	// We use the usable, default value of `DoUntilQuorumConfig` as ...
	var defaultQuorumConfig = ring.DoUntilQuorumConfig{}

[grobinson-grafana]

I think we can improve this comment, let's discuss offline.

[benclive]

As discussed offline, we will follow up with some better comments in another PR.

[grobinson-grafana]

Same here. Let's see if we can improve this comment. If I was unfamiliar with how this works, I would be surprised because results from all ingesters in all replication sets contradicts quorum by zone which by definition is a majority, not all.