Skip to content

Rollout-operator cannot call /ingester/prepare-downscale for clean scale down #18174

Description

@madaraszg-tulip

Describe the bug

I converted to zone-aware ingestion in my test loki cluster, and wanted to configure rollout-operator to call the /ingester/prepare-shutdown endpoint when scaling the statefulset down. The issue is that the statefulset has an incorrect serviceName, causing rollout-operator to not be able to talk to the pod.

serviceName: loki-testing-ingester-zone-a

There is no service called loki-testing-ingester-zone-a, the actual service created is loki-testing-ingester-zone-a-headless

To Reproduce
Steps to reproduce the behavior:

  1. Install Loki (v3.4.2) with zoneAwareReplication enabled
  2. Install rollout-operator with webhooks enabled
  3. Configure webhook for /admission/prepare-downscale
  4. Configure the ingester statefulsets with the necessary labels and annotations for rollout-operator
  5. Attempt to scale down an ingester statefulset

Expected behavior

I would expect the rollout-operator to be able to call the /ingester/prepare-shutdown endpoint on the pod

Environment:

  • Infrastructure: kubernetes
  • Deployment tool: helm (chart version 6.29.0)

Screenshots, Promtail config, or terminal output

Log from rollout-operator:

ts=2025-06-20T10:51:11.484391611Z method=admission.prepareDownscale() object.name=loki-testing-ingester-zone-a object.resource=statefulsets object.namespace=loki-testing request.dry_run=false request.uid=2a43b091-e246-4f53-9eb7-7b3a114fae69 request.gvk="autoscaling/v1, Kind=Scale" object.old_replicas=3 object.new_replicas=2 method=admission.PreparePodForShutdown url=loki-testing-ingester-zone-a-2.loki-testing-ingester-zone-a.loki-testing.svc.cluster.local:3100/ingester/prepare-shutdown index=2 method=POST level=error msg="error sending HTTP POST request" err="Post \"http://loki-testing-ingester-zone-a-2.loki-testing-ingester-zone-a.loki-testing.svc.cluster.local:3100/ingester/prepare-shutdown\": dial tcp: lookup loki-testing-ingester-zone-a-2.loki-testing-ingester-zone-a.loki-testing.svc.cluster.local on 172.20.0.10:53: no such host"

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions