feat(distributors): Use a pool of worker to push to ingesters.

[cyriltovena]

What this PR does / why we need it:

This introduce a pool to push to ingesters to avoid spawning a new goroutine each time and release go runtime scheduler.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[na--]

I am not 100% sure, but I think it might be possible for this to cause a panic 🤔 If we are in the middle of a Distributor.Push() call when the context gets cancelled, we might close d.ingesterTasks here right before we try to add to it here:

loki/pkg/distributor/distributor.go

Line 653 in 0738dd0

d.ingesterTasks <- pushIngesterTask{

[cyriltovena]

Change a bit the shutdown logic.

[chaudum]

lgtm