fix: Backport WAL replay hang fix to release-3.6.x (#21176)

[spiridonov]

What this PR does / why we need it:

Backports the WAL replay no-progress fix to release-3.6.x so it can be picked up by GEL 3.6 (which vendors this branch).

On 3.6.x, replayController.WithBackPressure has no exit condition: if a replay-triggered flush does not reduce in-memory bytes below 90% of replay_memory_ceiling (slow/unavailable storage backend, chunk_retain_period holding freshly flushed chunks, failed immediate flush ops being re-enqueued forever), the ingester spins in the backpressure loop indefinitely. It never finishes replay, never becomes ready, gets killed by the startup probe, and replays the same WAL again on the next start — an unrecoverable crash loop that only clearing the WAL breaks. We hit exactly this with a customer running GEL 3.6.10 (grafana/support-escalations#23469).

Two cherry-picks, in order:

• 347f2f7 refactor(ingestor): WAL Replay controller: Use singleflight to ensure only one inflight flush is happening (refactor(ingestor): WAL Replay controller: Use singleflight to ensure only one inflight flush is happening #19831)
• fb58ab8 fix: WAL replay hang when flush doesn't progress (fix: WAL replay hang when flush doesn't progress #21176)

The refactor is included because #21176 was written on top of it; taking both keeps replay_controller.go identical to main (both cherry-picks applied cleanly, no conflicts) instead of hand-porting the progress-detection logic onto the old sync.Cond implementation.

With this fix, a no-progress flush fails recovery with an explicit error (WAL replay flush made no progress: ... cannot recover) instead of hanging silently, and a zero replay_memory_ceiling no longer triggers backpressure on every record.

Special notes for your reviewer:

go test ./pkg/ingester/ -run TestReplayController passes on this branch, including the four tests added by #21176.

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 987b6f1. Configure here.}

[jnewbigin]

I'll cherry-pick #23631 into this PR