Description
Describe the bug
I converted to zone-aware ingestion in my test loki cluster, and wanted to configure rollout-operator to call the /ingester/prepare-shutdown endpoint when scaling the statefulset down. The issue is that the statefulset has an incorrect serviceName, causing rollout-operator to not be able to talk to the pod.
serviceName: loki-testing-ingester-zone-a
There is no service called loki-testing-ingester-zone-a
, the actual service created is loki-testing-ingester-zone-a-headless
To Reproduce
Steps to reproduce the behavior:
- Install Loki (v3.4.2) with zoneAwareReplication enabled
- Install rollout-operator with webhooks enabled
- Configure webhook for /admission/prepare-downscale
- Configure the ingester statefulsets with the necessary labels and annotations for rollout-operator
- Attempt to scale down an ingester statefulset
Expected behavior
I would expect the rollout-operator to be able to call the /ingester/prepare-shutdown endpoint on the pod
Environment:
- Infrastructure: kubernetes
- Deployment tool: helm (chart version 6.29.0)
Screenshots, Promtail config, or terminal output
Log from rollout-operator:
ts=2025-06-20T10:51:11.484391611Z method=admission.prepareDownscale() object.name=loki-testing-ingester-zone-a object.resource=statefulsets object.namespace=loki-testing request.dry_run=false request.uid=2a43b091-e246-4f53-9eb7-7b3a114fae69 request.gvk="autoscaling/v1, Kind=Scale" object.old_replicas=3 object.new_replicas=2 method=admission.PreparePodForShutdown url=loki-testing-ingester-zone-a-2.loki-testing-ingester-zone-a.loki-testing.svc.cluster.local:3100/ingester/prepare-shutdown index=2 method=POST level=error msg="error sending HTTP POST request" err="Post \"http://loki-testing-ingester-zone-a-2.loki-testing-ingester-zone-a.loki-testing.svc.cluster.local:3100/ingester/prepare-shutdown\": dial tcp: lookup loki-testing-ingester-zone-a-2.loki-testing-ingester-zone-a.loki-testing.svc.cluster.local on 172.20.0.10:53: no such host"