Skip to content

[loki] Add preserveReplicas option for KEDA migration safety #652

Description

@skoenig

Problem

The Loki Helm chart supports KEDA autoscaling via kedaAutoscaling, but it does not provide a safe migration path from a statically scaled Deployment to a KEDA-managed one.

When kedaAutoscaling.enabled: true is set on a component (e.g. querier), the chart removes .spec.replicas from the rendered Deployment manifest via this guard in templates/_workload.tpl:

{{- if and (not (dig "autoscaling" "enabled" false $component)) (not (dig "kedaAutoscaling" "enabled" false $component)) (not (kindIs "invalid" $component.replicas)) }}
  replicas: {{ ... }}
{{- end }}

Because Helm 3 uses a three-way merge, removing replicas from the manifest causes Kubernetes to reset .spec.replicas to the default of 1 at the moment the new release is applied. KEDA's HPA then needs at least one pollingInterval to fetch metrics and scale back up. During that window a production querier deployment can drop from, for example, 40 replicas to 1, causing a brief outage.

Prior art in the Grafana ecosystem

The same problem was identified and solved in the Grafana Mimir Helm chart:

Mimir's fix adds a preserveReplicas flag per component. The Deployment template keeps .spec.replicas in the manifest when both KEDA is enabled and preserveReplicas: true, allowing the HPA to take over without the deployment ever falling back to 1 replica:

{{- if or (not .Values.distributor.kedaAutoscaling.enabled) (.Values.distributor.kedaAutoscaling.preserveReplicas) }}
  replicas: {{ .Values.distributor.replicas }}
{{- end }}

The documented migration is then:

  1. Enable KEDA with preserveReplicas: true.
  2. Wait until the HPA is active and managing replicas.
  3. Remove preserveReplicas in a follow-up change.

Proposed solution

Add the same preserveReplicas option to the Loki chart's kedaAutoscaling block for every workload component, and update templates/_workload.tpl to keep .spec.replicas when it is set. For example:

{{- if and (not (dig "autoscaling" "enabled" false $component)) (or (not (dig "kedaAutoscaling" "enabled" false $component)) (dig "kedaAutoscaling" "preserveReplicas" false $component)) (not (kindIs "invalid" $component.replicas)) }}
  replicas: {{ if eq $target "single-binary" }}{{ include "loki.monolithicReplicas" $ctx }}{{ else }}{{ $component.replicas }}{{ end }}
{{- end }}

Also add the field to values.yaml for each component that supports kedaAutoscaling (e.g. querier.kedaAutoscaling.preserveReplicas: false).

Impact

Without this, users moving production Loki clusters to KEDA autoscaling risk a brief but severe scale-down of critical components during the migration. This makes the feature effectively unsafe for GitOps-driven, zero-downtime environments.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions