Fix TIME_WAIT connection leak from queriers to compactor

[MasslessParticle]

http://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/

This case is overwhelmingly likely to be from the connection pool as DefaultMaxIdleConnsPerHost is only 2. This PR also makes sure to drain the http response body, just in case.

[DylanGuedes]

lgtm. btw, first time seeing this ioutil.discard, interesting

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
-               loki	-0.2%

[grafanabot]

The backport to release-2.6.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-6681-to-release-2.6.x origin/release-2.6.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 25bd0a4a55d84594df74a5c27e02ecab90130cd1
# Push it to GitHub
git push --set-upstream origin backport-6681-to-release-2.6.x
git switch main
# Remove the local backport branch
git branch -D backport-6681-to-release-2.6.x

Then, create a pull request where the base branch is release-2.6.x and the compare/head branch is backport-6681-to-release-2.6.x.

[dannykopping]

Nice! I remember debugging this exact issue with Peter from Mimir last year on the billing forwarder:
https://github.com/grafana/backend-enterprise/pull/1472/files#diff-0e057ea5ec711488359552e50ba11ed62dceb0884abfa6c3c8654a82e7246080R174

PS the backporting seems to have failed?