feat: Improve pattern ingester tracing

[trevorwhitney]

What this PR does / why we need it:

We had to turn off aggregated metrics in a prod cluster because the service_name label being used for __aggregated_metric__ had crazy cardinality, it was some huge JSON blob. This PR adds some more tracing and log signals to the aggregated metrics path through the pattern ingester so we can debug what's going on.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[loki-gh-app]

This PR must be merged before a backport PR will be created.

[benclive]

LGTM. I added a couple of small suggestions but I don't think the impact is large so I'm happy to approve to unblock.

[benclive]

Should the "services" be limited somehow? I'm not expecting a lot but just in case there is a large number of services we're trying to attach to a span.

[benclive]

Same here, it shouldn't be big but it might be worth checking the length of this just in case?

[loki-gh-app]

The backport to k226 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-14707-to-k226 origin/k226
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 80aec2548203957dbb834ba69e6d734d9054416d

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-14707-to-k226
# Create the PR body template
PR_BODY=$(gh pr view 14707 --json body --template 'Backport 80aec2548203957dbb834ba69e6d734d9054416d from #14707{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title 'feat: Improve pattern ingester tracing (backport k226)' --body-file - --label 'size/L' --label 'product-approved' --label 'backport' --base k226 --milestone k226 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-14707-to-k226

# Create a pull request where the `base` branch is `k226` and the `compare`/`head` branch is `backport-14707-to-k226`.

# Remove the local backport branch
git switch main
git branch -D backport-14707-to-k226