fix(healthcheck): read shm status so workers do not keep unhealthy nodes - #13894
Open
BetterAndBetterII wants to merge 1 commit into
Open
fix(healthcheck): read shm status so workers do not keep unhealthy nodes#13894BetterAndBetterII wants to merge 1 commit into
BetterAndBetterII wants to merge 1 commit into
Conversation
Active checks write health to shared memory, then notify peers over resty.events. Routing still used get_target_status(), which only reads the worker-local cache, and the balancer picker was keyed on status_ver (bumped only on a local event). A worker that missed the event kept sending traffic to a node already marked unhealthy in shm. Consult shm in fetch_node_status and include shm health in the picker cache key so a shm-only flip rebuilds the picker. Fixes apache#13888
dosubot
Bot
added
size:L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Active health checks write the real target state to shared memory, then notify other nginx workers over
resty.events. Routing still usedget_target_status(), which only reads that worker-local cache, and the balancer picker was keyed onchecker.status_ver(bumped only when this worker receives the event).A worker that missed the event (or whose checker was created after another worker already flipped the node) kept sending traffic to a node that was already unhealthy in shm — for minutes in multi-worker deployments with priority failover.
This change:
fetch_node_status(falling back to the local cache when shm has no entry yet).Checklist
Fixes #13888