Description
Is your feature request related to a problem? Please describe.
The current Loki Helm chart's ServiceMonitor does not include a jobLabel. This results in scraped metrics having a job label formatted as "namespace/loki-component".
However, the grafana/loki mixin defines alerts and dashboards that expect the job label to be "namespace/component", which does not match with the metrics scraped by the Helm chart's default ServiceMonitor configuration.
I understand that you advise using the metamonitoring helm chart, but our preference is to integrate Loki with our current k8s-monitoring
stack configured to scrape based on prometheus-operator CRDs, and we don't need a dedicated cluster for metamonitoring or to push metrics remotely.
Describe the solution you'd like
It would be interesting to allow the override of jobLabel selectors within the config.libsonnet file of the Loki mixin. This would provide the flexibility to align the job label with different configurations. A similar approach is successfully implemented in the Grafana Mimir mixin, as seen here.
Describe alternatives you've considered
A simpler alternative could be to allow setting a custom jobLabel directly in the ServiceMonitor template within the Helm chart. This would enable us to set the job to the value the mixin expect. However, I think having the job named "namespace/loki-component" is good, it avoids having the same label between loki and for example mimir, and the ability to customize the job selector in config.libsonnet would allow more flexibility.
Additional context
I can submit a PR to add the jobLabel to the ServiceMonitor if that's ok for you.
I can also look into the mixin changes, but that's a more substanntial taks and I'm not sure to have the time for it at this moment.