Problem
The Loki Helm chart supports KEDA autoscaling via kedaAutoscaling, but it does not provide a safe migration path from a statically scaled Deployment to a KEDA-managed one.
When kedaAutoscaling.enabled: true is set on a component (e.g. querier), the chart removes .spec.replicas from the rendered Deployment manifest via this guard in templates/_workload.tpl:
{{- if and (not (dig "autoscaling" "enabled" false $component)) (not (dig "kedaAutoscaling" "enabled" false $component)) (not (kindIs "invalid" $component.replicas)) }}
replicas: {{ ... }}
{{- end }}
Because Helm 3 uses a three-way merge, removing replicas from the manifest causes Kubernetes to reset .spec.replicas to the default of 1 at the moment the new release is applied. KEDA's HPA then needs at least one pollingInterval to fetch metrics and scale back up. During that window a production querier deployment can drop from, for example, 40 replicas to 1, causing a brief outage.
Prior art in the Grafana ecosystem
The same problem was identified and solved in the Grafana Mimir Helm chart:
Mimir's fix adds a preserveReplicas flag per component. The Deployment template keeps .spec.replicas in the manifest when both KEDA is enabled and preserveReplicas: true, allowing the HPA to take over without the deployment ever falling back to 1 replica:
{{- if or (not .Values.distributor.kedaAutoscaling.enabled) (.Values.distributor.kedaAutoscaling.preserveReplicas) }}
replicas: {{ .Values.distributor.replicas }}
{{- end }}
The documented migration is then:
- Enable KEDA with
preserveReplicas: true.
- Wait until the HPA is active and managing replicas.
- Remove
preserveReplicas in a follow-up change.
Proposed solution
Add the same preserveReplicas option to the Loki chart's kedaAutoscaling block for every workload component, and update templates/_workload.tpl to keep .spec.replicas when it is set. For example:
{{- if and (not (dig "autoscaling" "enabled" false $component)) (or (not (dig "kedaAutoscaling" "enabled" false $component)) (dig "kedaAutoscaling" "preserveReplicas" false $component)) (not (kindIs "invalid" $component.replicas)) }}
replicas: {{ if eq $target "single-binary" }}{{ include "loki.monolithicReplicas" $ctx }}{{ else }}{{ $component.replicas }}{{ end }}
{{- end }}
Also add the field to values.yaml for each component that supports kedaAutoscaling (e.g. querier.kedaAutoscaling.preserveReplicas: false).
Impact
Without this, users moving production Loki clusters to KEDA autoscaling risk a brief but severe scale-down of critical components during the migration. This makes the feature effectively unsafe for GitOps-driven, zero-downtime environments.
Problem
The Loki Helm chart supports KEDA autoscaling via
kedaAutoscaling, but it does not provide a safe migration path from a statically scaled Deployment to a KEDA-managed one.When
kedaAutoscaling.enabled: trueis set on a component (e.g.querier), the chart removes.spec.replicasfrom the rendered Deployment manifest via this guard intemplates/_workload.tpl:{{- if and (not (dig "autoscaling" "enabled" false $component)) (not (dig "kedaAutoscaling" "enabled" false $component)) (not (kindIs "invalid" $component.replicas)) }} replicas: {{ ... }} {{- end }}Because Helm 3 uses a three-way merge, removing
replicasfrom the manifest causes Kubernetes to reset.spec.replicasto the default of 1 at the moment the new release is applied. KEDA's HPA then needs at least onepollingIntervalto fetch metrics and scale back up. During that window a production querier deployment can drop from, for example, 40 replicas to 1, causing a brief outage.Prior art in the Grafana ecosystem
The same problem was identified and solved in the Grafana Mimir Helm chart:
preserveReplicasoption for kedaAutoscaling to preservereplicaseven if keda is enabledMimir's fix adds a
preserveReplicasflag per component. The Deployment template keeps.spec.replicasin the manifest when both KEDA is enabled andpreserveReplicas: true, allowing the HPA to take over without the deployment ever falling back to 1 replica:{{- if or (not .Values.distributor.kedaAutoscaling.enabled) (.Values.distributor.kedaAutoscaling.preserveReplicas) }} replicas: {{ .Values.distributor.replicas }} {{- end }}The documented migration is then:
preserveReplicas: true.preserveReplicasin a follow-up change.Proposed solution
Add the same
preserveReplicasoption to the Loki chart'skedaAutoscalingblock for every workload component, and updatetemplates/_workload.tplto keep.spec.replicaswhen it is set. For example:{{- if and (not (dig "autoscaling" "enabled" false $component)) (or (not (dig "kedaAutoscaling" "enabled" false $component)) (dig "kedaAutoscaling" "preserveReplicas" false $component)) (not (kindIs "invalid" $component.replicas)) }} replicas: {{ if eq $target "single-binary" }}{{ include "loki.monolithicReplicas" $ctx }}{{ else }}{{ $component.replicas }}{{ end }} {{- end }}Also add the field to
values.yamlfor each component that supportskedaAutoscaling(e.g.querier.kedaAutoscaling.preserveReplicas: false).Impact
Without this, users moving production Loki clusters to KEDA autoscaling risk a brief but severe scale-down of critical components during the migration. This makes the feature effectively unsafe for GitOps-driven, zero-downtime environments.