feat(canary): Support passing arbitrary set of labels to use for the query

[iNoahNothing]

What this PR does / why we need it:
Loki Canary logs are collected and parsed by logging agents whose configuration is controlled by users. Since Loki does not parse logs directly, the set of labels for each log line is up to users and extremely variable.

The current approach only allows for two labels to be used for the query sent to Loki which can impact the canary's ability to find the logs it is looking for. By adding a -labels flag, we are able to pass an arbitrary set of labels to be used in the query. Combined with -query-append, you can craft a full LogQL query to correctly and efficiently return the logs it is looking for.

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[CLAassistant]

All committers have signed the CLA.

[iNoahNothing]

@MasslessParticle Sorry for the ping but you seem to be the last person whose looked at the canary stuff #14287

Is there anything I can do to get this over the line?

[iNoahNothing]

@paul1r Sorry for the ping. Any chance we can get eyes on this or should I expect to need to fork the project to make this change?

[slim-bean]

minor nit but would prefer we only do the strings.Split one time

[slim-bean]

it might also be nice to put a little bit of error handling here. If someone misconfigures the string they will get a panic and an index out of range exception.

we could do a quick check and if the length of the array after split !=2 return an error (not sure how the caller handles errors) or we could still panic but with a very clear message

[iNoahNothing]

Great call out. This was bad pattern and I definitely should have been doing this minimal amount of error handling for the parsing logic.

[slim-bean]

sorry this one didn't get any attention sooner.

just a couple suggestions.

I think having a simple parser for this config is fine, I wouldn't expect people to want to add really complex label values for this use case, however I threw a little more text in the argument description to clarify this.

Also a request to change the handling of the string a little and maybe add a little error handling to help folks if they do misconfigure the label string

[iNoahNothing]

Thanks for the review @slim-bean and sorry for the delay in getting back to this! Appreciate the additions to callout the very simple parsing logic that is being done. Agreed that I do not expect the labels to be complex at all for this use case. Really just wanted to open up a way to add more label filtering when someone is using a different labeling scheme.

Let me know if there is anything more you need from me. Cheers!

[slim-bean]

LGTM Thanks!