Skip to content

feat(compactor HS): use streaming RPCs for interacting with job queue over grpc - #18125

Merged
sandeepsukhani merged 6 commits into
mainfrom
compactor-job-queue-streaming-rpc
Jun 17, 2025
Merged

feat(compactor HS): use streaming RPCs for interacting with job queue over grpc#18125
sandeepsukhani merged 6 commits into
mainfrom
compactor-job-queue-streaming-rpc

Conversation

@sandeepsukhani

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
The PR actually contains multiple changes:

  1. Move the proto definition for job queue RPCs to the existing compactor client protos. This would allow us to reuse the existing compactor client and its configs.
  2. I changed RPCs to streaming RPCs for interacting with the job queue over grpc to avoid polling for new jobs. I took some inspiration from our existing code for query processing using query-frontend/scheduler and queries.
  3. I have made the job builder loop not return an error when it sees a failure in building the job. It will continue to attempt job building with the next tick event. We will add a metric to track the failures and alert when multiple consecutive failed attempts occur.

Checklist

  • Tests updated
@sandeepsukhani
sandeepsukhani requested a review from a team as a code owner June 17, 2025 10:34
@sandeepsukhani
sandeepsukhani merged commit 385e317 into main Jun 17, 2025
64 checks passed
@sandeepsukhani
sandeepsukhani deleted the compactor-job-queue-streaming-rpc branch June 17, 2025 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 participants