Skip to content

perf(pd): reduce disaggregated serving latency - #1481

Open
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:perf/pd-reduce-disaggregated-latency
Open

perf(pd): reduce disaggregated serving latency#1481
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:perf/pd-reduce-disaggregated-latency

Conversation

@sufubao

@sufubao sufubao commented Aug 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • make detokenization and PD KV polling intervals configurable through environment variables
  • reduce router-loop overhead by making metric gauge cadence configurable and avoiding uvloop busy-yields for sub-millisecond intervals

The request shared-memory handle cleanup was extracted into #1482.

Tests

  • pre-commit run black --files
  • pre-commit run flake8 --files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant

@sufubao