fix: handle INIT from frontend in running scheduler which is not in shouldRun state - #19866
Conversation
spiridonov
left a comment
There was a problem hiding this comment.
Thank you for tackling that bug @mxab !
|
|
||
| // not_running -> running, shouldRun == true | ||
| // to simulate last "setRunState(true)" happening after FrontendLoop started | ||
| s.setRunState(true) |
There was a problem hiding this comment.
Correct me if I'm wrong. This call does not effectively change anything in the test results or in the mock since FrontendLoop func has already returned by that moment, right?
There was a problem hiding this comment.
Ok If i remember it correctly my intention was to "simulate" the rest of the behaviour in case something happens in the future.
But yes doesn't make to much sense. I removed the setRunState(true) and the assertion
remove unnecessary state changes in tests
|
@mxab thank you! |
|
Great to get this merged. Btw I previously opened a more complex pr that introduced a general failsafe from the frontend side. I assumed it was to big and therefore ignored. Would you mind sharing your opinion on that one as well? |
… `shouldRun` state (#19866) Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro> Co-authored-by: Stas Spiridonov <stanislav@spiridonov.pro>
Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro> Co-authored-by: Max Fröhlich <maxbruchmann@gmail.com>
… `shouldRun` state (#19866) Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro> Co-authored-by: Stas Spiridonov <stanislav@spiridonov.pro>
Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro> Co-authored-by: Max Fröhlich <maxbruchmann@gmail.com>

What this PR does / why we need it:
We expirience the situation where the scheduler is not yet in a "shouldRun" state but receives an init message from the frontend.
In this case the scheduler will not reply with an
OKbut rather will just wait for the next messageThis means the frontend worker is then blocked as it waits for any kind of reply from the scheduler.
Therefore the frontend worker connects to the scheduler but does not forward incoming queries
Which issue(s) this PR fixes:
Fixes #19528
This PR mitigates this issue by sending as the scheduler now sends an
ERRORin case it's RUNNING but shouldRun is still falseThis PR is the "smaller" version of this PR as I hope its easier to review while the other one also add an additional timeout on the frontend side
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mddeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PRNote
Medium Risk
Medium risk because it changes the scheduler/frontend handshake behavior in
FrontendLoop, which can affect query routing and reconnect behavior during ring transitions, though the change is small and covered by a new test.Overview
Fixes a frontend connection stall by explicitly responding to
INITwhen the scheduler service isRunningbutshouldRunis still false (eg, not yet in the ring replication set).Instead of silently proceeding without an
INITresponse, the scheduler now logs and sendsschedulerpb.ERRORso the frontend can immediately retry a different scheduler. Adds a focused unit test and enhances the frontend loop mock to support scriptedRecv()behavior.Reviewed by Cursor Bugbot for commit 8320a29. Bugbot is set up for automated code reviews on this repo. Configure here.