fix(helm): Use correct serviceName in zone-aware ingester statefulsets

[meinenec]

What this PR does / why we need it:

When running with zone-aware ingesters, the chart creates three services for loki-testing-ingester-zone-<zone a-c>-headless but when rollout-operator attempts to call /ingester/prepare-downscale it gets serviceName: loki-testing-ingester-zone-a from the statefulset. This does not have the -headless suffix and the call fails. This causes the ingesters to fail to scale down properly.

Which issue(s) this PR fixes:
Fixes #18174

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[jkroepke]

LGTM, just one nit.

[meinenec]

LGTM, just one nit.

apologies for missing that and thank you for reviewing!

[jkroepke]

LGTM

[meinenec]

@jkroepke I was introducing some conflicts while rebasing to update my fork. I'll stop doing that and I have addressed new changelog issues. Apologies for the noise on this PR.

[jkroepke]

Thank you!

LGTM

[jkroepke]

For record: this PR breaks existing setups:

Error: UPGRADE FAILED: cannot patch "loki-ingester-zone-a" with kind StatefulSet: StatefulSet.apps "loki-ingester-zone-a" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

We are currently discuss, how we handle this. (extensive note in the CHANGELOG OR revert of the PR)

[abenbachir]

We pick-up latest chart 6.34.0 and got that same bug. Is it possible to provide us with a fix in 6.35.0 since we need to use some feature in latest chart.

[jkroepke]

@abenbachir Quick Workaround is delete the Statefulset manually before running helm upgrade.

[Jayclifford345]

Hey all, so we have two options here, and we are happy to take a vote on which you would prefer:
👍 - We include the PR and those using zone aware ingesters manually delete their ingesters and perform the upgrade. No data will be lost, because it lives inside the PersistentVolumeClaim, not on the StatefulSet object. We would update the change log to reflect this known issue with upgrading

👎 - We move to removing this PR which fixes the serviceName in the zone-aware ingesters and consider fixing it when we move to Loki 4.0 helm chart in the future.

[abenbachir]

I can do the workaround, but how do we know you guys won't rollback this change in next release and we will be doing another workaround.

Also can you provide excat kubctl commands to delete the ingesters, we would like to try it in our staging env.

[jkroepke]

The mentioned workaround is now part of the official upgrading documentation: https://grafana.com/docs/loki/next/setup/upgrade/upgrade-to-6x/#breaking-zone-aware-ingester-statefulset-servicename-fix-6340