Skip to content

fix(agents): evict dead warmed processes - #2388

Open
kokoite wants to merge 1 commit into
livekit:mainfrom
kokoite:fix/evict-dead-warmed-process
Open

fix(agents): evict dead warmed processes#2388
kokoite wants to merge 1 commit into
livekit:mainfrom
kokoite:fix/evict-dead-warmed-process

Conversation

@kokoite

@kokoite kokoite commented Aug 31, 2026

Copy link
Copy Markdown

Description

Fixes #2321.

ProcPool transferred ownership of a warmed process's MultiMutex unlock token to warmedProcQueue, but the watcher only removed the executor from executors when that idle process exited. The stale queue entry could therefore be dequeued for a later job, and its capacity slot remained occupied until then.

Changes Made

  • Retain the exact warmed queue entry in procWatchTask and remove it by object identity if the process exits before a consumer claims it.
  • Release the associated process slot when the watcher removes a dead queued entry.
  • Drain queued entries before shutdown cleanup so close() and the watcher cannot release the same slot twice.
  • Add deterministic race coverage for process exit before dequeue, dequeue before exit, shutdown while queued, and preservation of other healthy entries.

Pre-Review Checklist

  • Build passes: Build, lint, typecheck, formatting, tests, and Throws validation pass locally
  • AI-generated code reviewed: Removed unnecessary comments and ensured code quality
  • Changes explained: All changes are documented and justified above
  • Scope appropriate: All changes address the warmed-process queue lifecycle
  • Video demo: Not applicable; this is an internal process-pool race covered deterministically by unit tests

Testing

  • Automated tests added/updated
  • pnpm test agents/src/ipc/proc_pool.test.ts — 8 passed
  • pnpm test agents --silent — 2,322 passed, 5 skipped
  • pnpm build:agents
  • pnpm typecheck
  • pnpm lint
  • pnpm format:check
  • pnpm throws:check
  • restaurant_agent.ts and realtime_agent.ts manual checks are not applicable to this internal process lifecycle fix

Additional Notes

The ownership rule is based on synchronous queue removal: launchJob(), watcher cleanup, and close() each release a token only after successfully claiming its queue entry. This keeps every warmed-process capacity slot paired with exactly one release path.


Note to reviewers: Please ensure the pre-review checklist is completed before starting your review.

@kokoite
kokoite requested a review from a team as a code owner August 31, 2026 10:45
@changeset-bot

changeset-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 87f4d75

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-azure Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-krisp Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-protoface Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-soniox Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugins-test Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@CLAassistant

CLAassistant commented Aug 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants