feat: implement volume-based pattern filtering

[trevorwhitney]

What this PR does / why we need it:

Adds volume-based filtering to the pattern ingester to persist only the most significant patterns based on log volume. This helps reduce storage and processing overhead by filtering out low-volume patterns.

Key changes:

• Add Volume and SampleCount tracking to LogCluster to measure pattern significance
• Add VolumeThreshold configuration (default 0.99) to control filtering percentage
• Implement filterClustersByVolume function to select top X% of patterns by volume
• Integrate volume filtering into the prune() method of pattern streams
• Add comprehensive tests for volume filtering functionality
• Remove service_name from pattern metrics due to cardinality issues
• Remove histograms for capturing pattern/agg metric payloads (again because of cardinality issues)

The volume threshold represents the percentage of total log volume to retain. For example, a threshold of 0.8 keeps only the patterns that represent the top 80% of log volume, filtering out the long tail of low-volume patterns.

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[github-actions]

💻 Deploy preview deleted.

[trevorwhitney]

Tested in ops over the past few days. Overall it seems to be reducing the number of patterns written. There are still spikes, and we see at the end of this time period there was a spike as well, but overall I think this is worth merging.

Here's 7 days of patterns on ops