Skip to content

fix: handle INIT from frontend in running scheduler which is not in shouldRun state - #19866

Merged
spiridonov merged 13 commits into
grafana:mainfrom
mxab:fix/scheduler-init-response
Apr 6, 2026
Merged

fix: handle INIT from frontend in running scheduler which is not in shouldRun state#19866
spiridonov merged 13 commits into
grafana:mainfrom
mxab:fix/scheduler-init-response

Conversation

@mxab

@mxab mxab commented Nov 14, 2025

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

We expirience the situation where the scheduler is not yet in a "shouldRun" state but receives an init message from the frontend.
In this case the scheduler will not reply with an OK but rather will just wait for the next message

This means the frontend worker is then blocked as it waits for any kind of reply from the scheduler.
Therefore the frontend worker connects to the scheduler but does not forward incoming queries

Which issue(s) this PR fixes:

Fixes #19528

This PR mitigates this issue by sending as the scheduler now sends an ERROR in case it's RUNNING but shouldRun is still false

This PR is the "smaller" version of this PR as I hope its easier to review while the other one also add an additional timeout on the frontend side

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Note

Medium Risk
Medium risk because it changes the scheduler/frontend handshake behavior in FrontendLoop, which can affect query routing and reconnect behavior during ring transitions, though the change is small and covered by a new test.

Overview
Fixes a frontend connection stall by explicitly responding to INIT when the scheduler service is Running but shouldRun is still false (eg, not yet in the ring replication set).

Instead of silently proceeding without an INIT response, the scheduler now logs and sends schedulerpb.ERROR so the frontend can immediately retry a different scheduler. Adds a focused unit test and enhances the frontend loop mock to support scripted Recv() behavior.

Reviewed by Cursor Bugbot for commit 8320a29. Bugbot is set up for automated code reviews on this repo. Configure here.

@mxab
mxab requested a review from a team as a code owner November 14, 2025 10:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.

Comment thread pkg/scheduler/scheduler.go Outdated

@spiridonov spiridonov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for tackling that bug @mxab !

Comment thread pkg/scheduler/scheduler.go Outdated
Comment thread pkg/scheduler/scheduler_test.go Outdated

// not_running -> running, shouldRun == true
// to simulate last "setRunState(true)" happening after FrontendLoop started
s.setRunState(true)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong. This call does not effectively change anything in the test results or in the mock since FrontendLoop func has already returned by that moment, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok If i remember it correctly my intention was to "simulate" the rest of the behaviour in case something happens in the future.
But yes doesn't make to much sense. I removed the setRunState(true) and the assertion

mxab and others added 4 commits April 4, 2026 23:25
remove unnecessary state changes in tests
Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro>
@spiridonov
spiridonov merged commit 6f81a0c into grafana:main Apr 6, 2026
84 checks passed
@spiridonov

Copy link
Copy Markdown
Contributor

@mxab thank you!

@mxab

mxab commented Apr 6, 2026

Copy link
Copy Markdown
Contributor Author

Great to get this merged. Btw I previously opened a more complex pr that introduced a general failsafe from the frontend side. I assumed it was to big and therefore ignored.

Would you mind sharing your opinion on that one as well?

@spiridonov

#19540

spiridonov added a commit that referenced this pull request May 21, 2026
… `shouldRun` state (#19866)

Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro>
Co-authored-by: Stas Spiridonov <stanislav@spiridonov.pro>
spiridonov added a commit that referenced this pull request May 21, 2026
Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro>
Co-authored-by: Max Fröhlich <maxbruchmann@gmail.com>
spiridonov added a commit that referenced this pull request May 21, 2026
… `shouldRun` state (#19866)

Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro>
Co-authored-by: Stas Spiridonov <stanislav@spiridonov.pro>
spiridonov added a commit that referenced this pull request May 21, 2026
Signed-off-by: Stas Spiridonov <stanislav@spiridonov.pro>
Co-authored-by: Max Fröhlich <maxbruchmann@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment