feat: add state change lock mechanism

[Segflow]

What this PR does:

This PR adds the ability to lock partition state change. If a state change is locked we cannot change the state of a partition ring.

This is useful if we want to deactivate a partition ring and prevent it from being activated again.

We also add these 2 metrics:

• partition_ring_partition_state_change_locked{name, partition_id, state}
• partition_ring_partition_state_change_locked_timestamp_seconds{name, partition_id, state}

Checklist

• Tests updated

---

Note

Introduce a lock to prevent partition state changes, updating proto/model, editor, lifecycler, HTTP API/UI, watcher logging, and tests.

• Ring Model/Proto:
  • Add PartitionDesc fields: stateChangeLocked and stateChangeLockedTimestamp (partition_ring_desc.proto + regenerated pb.go).
  • Enforce lock in UpdatePartitionState() (now returns (bool, error)) and add UpdatePartitionStateChangeLock().
  • Merge logic propagates lock fields based on timestamp.
• Editor/API:
  • Add SetPartitionStateChangeLock() and helper SetPartitionStateChangeLock(...).
  • changePartitionState() and callers adapted to new return signature.
• HTTP/UI:
  • Surface stateChangeLocked in page data; disable change buttons when locked.
  • New POST actions: state_change_lock (lock/unlock) and change_state_and_lock (change then lock).
• Lifecycler:
  • Reconcile path updated to use new UpdatePartitionState signature.
• Watcher:
  • Log partitions with locked state changes on updates.
• Tests:
  • Add/extend tests for lock behavior across model, editor, HTTP handler, watcher, and ring tests; update existing calls for new signatures.

^{Written by Cursor Bugbot for commit fd822d0. This will update automatically on new commits. Configure here.}

[CLAassistant]

All committers have signed the CLA.

[cursor]

Bug: State lock bypassed by automatic partition reconciliation

The new state change lock mechanism only protects state changes through changePartitionState, but reconcileOwnedPartition directly calls ring.UpdatePartitionState() bypassing the lock check. This means even when a partition has StateChangeLocked set to true, the automatic Pending → Active transition during reconciliation will still occur, defeating the purpose of the lock mechanism.

ring/partition_instance_lifecycler.go#L367-L368

dskit/ring/partition_instance_lifecycler.go

Lines 367 to 368 in e55c331

	level.Info(l.logger).Log("msg", "switching partition state because enough owners have been registered and minimum waiting time has elapsed", "partition", l.cfg.PartitionID, "from_state", PartitionPending, "to_state", PartitionActive)
	return ring.UpdatePartitionState(partitionID, PartitionActive, now), nil

ring/partition_ring_editor.go#L70-L73

dskit/ring/partition_ring_editor.go

Lines 70 to 73 in e55c331

	if partition.StateChangeLocked {
	return false, ErrPartitionStateChangeLocked
	}

 

---

[cursor]

Bug: Non-atomic state change and lock creates race condition

The change_state_and_lock action performs two separate CAS operations (ChangePartitionState followed by LockPartitionStateChange) that are not atomic. Between these two operations, another concurrent request could change the partition state, resulting in locking an unintended state. The intent is to atomically change state and lock, but a concurrent state change between lines 165-173 could lead to the wrong state being locked.

 

[cursor]

Bug: Lifecycler bypasses state lock during automatic reconciliation

The reconcileOwnedPartition function calls ring.UpdatePartitionState() directly without checking the StateChangeLocked flag. This means when a partition is locked and its state transitions automatically from Pending to Active (when enough owners join), the lock is bypassed. The PR adds state lock enforcement in changePartitionState via ChangePartitionState, but this automatic reconciliation path doesn't use that function, defeating the purpose of the lock for Pending→Active transitions.

ring/partition_instance_lifecycler.go#L365-L368

dskit/ring/partition_instance_lifecycler.go

Lines 365 to 368 in 56874a1

	// have been added since more than the waiting period.
	if partition.IsPending() && ring.PartitionOwnersCountUpdatedBefore(partitionID, now.Add(-l.cfg.WaitOwnersDurationOnPending)) >= l.cfg.WaitOwnersCountOnPending {
	level.Info(l.logger).Log("msg", "switching partition state because enough owners have been registered and minimum waiting time has elapsed", "partition", l.cfg.PartitionID, "from_state", PartitionPending, "to_state", PartitionActive)
	return ring.UpdatePartitionState(partitionID, PartitionActive, now), nil

 

---

[cursor]

Bug: State lock bypassed in automatic partition reconciliation

The reconcileOwnedPartition function directly calls ring.UpdatePartitionState() to automatically transition partitions from Pending to Active when enough owners register. This bypasses the new StateChangeLocked check that was added in changePartitionState(). As a result, even if an operator locks a partition to prevent state changes, the automatic lifecycle reconciliation will still change a Pending partition to Active, defeating the purpose of the lock mechanism for this transition path.

ring/partition_instance_lifecycler.go#L367-L368

dskit/ring/partition_instance_lifecycler.go

Lines 367 to 368 in 84605bc

	level.Info(l.logger).Log("msg", "switching partition state because enough owners have been registered and minimum waiting time has elapsed", "partition", l.cfg.PartitionID, "from_state", PartitionPending, "to_state", PartitionActive)
	return ring.UpdatePartitionState(partitionID, PartitionActive, now), nil

 

---

[pracucci]

The changes LGTM. I'm not convinced the new metrics are required. I think logs better fit this use case.

[pracucci]

Does the onclick here prevents the change if canceled? I'm wondering if the safest option would just be creating a <form> for each different action (in this case it would be 1 more form).

[Segflow]

Yes the onlick prevent the change if cancelled, the browser will just not send the request.

[pracucci]

These metrics are going to be high cardinality because exposed for each partition by each pod. Are they really required? Looks like useful only once in a lifetime (maybe when investigating an incident) and maybe can be replaced by a log when the lock change? It could be logged here by the watcher.

[Segflow]

I preferred metrics so I can have a dashboard likes the one we have for flux ignore dashboard
But since this is only an emergency lever, I agree that this high cardinality metric is not necessary, I replaced it with a simple warn log:

level.Warn(w.logger).Log("msg", "partition state change is locked", "partition_id", partitionIDStr, "partition_state", state)

[cursor]

Bug: Warning logs emitted on every ring update

The warning log for locked partition state changes is emitted every time updatePartitionRing is called, which happens on every ring update from the KV store watcher. For long-lived locked partitions in busy clusters with frequent ring updates, this will cause significant log spam. The intent appears to be logging when the lock changes, not on every ring update. Consider tracking previously locked partitions and only logging when the lock state actually changes.

 

[pracucci]

This is extremely verbose. Why not just logging when it has changed compared to the oldRing?