Introduce a secondary grouping key for rollout operation

[boldfield]

The way rollout-operator currently is designed in percludes deploying all of the LGTM stack to a single namespace (which is required in some circumstances). This adds an optional additional label that can be used to employ a secondary grouping to allow for all components to be deployed in a single namespace and have the rollout-operator execute as expected. For example:

Loki ingester:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: loki-ingester-zone-a
    rollout-group: ingester
    rollout-secondary-group: loki

Mimir ingester:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: mimir-ingester-zone-a
    rollout-group: ingester
    rollout-secondary-group: mimir

[56quarters]

Thanks for this PR but unfortunately we (Mimir team) don't support or test running Mimir in the same namespace as Loki or Tempo. I think we're pretty unlikely to want to support this. If you really need to run the databases in the same namespace it may be enough to change the rollout-group of each database to make them unique (e.g. mimir-ingester, loki-ingester, etc). Can you expand a bit on why this is required?

[boldfield]

@56quarters My team runs in a restricted environment where we are required to deploy all of our k8s workloads in a single namespace. We have been using the approach you suggested with customized helm charts for loki and tempo and have PRs up for both. The tempo PR was merged (grafana/loki#15063) but I'm getting pushback on the Loki PR (grafana/helm-charts#3443) and devised this as an alternative approach which would negate the requirement for a helm chart change.

[56quarters]

Oof, that's unfortunate. This isn't something we're going to support in the rollout-operator so I think the helm chart approach is your best bet if you can't use a separate namespaces for each database. Maybe kustomize would allow you to modify the selectors in the helm charts if their respective maintainers don't want to support this either (apologies if you've already looked into that).

[boldfield]

we were previously using kustomize but unfortunately that has become a non-option in our environment as well 👎

Thanks for your consideration.