feat: enable KV cache events on vLLM inference pods by default

[andyzhangx]

Description

Enable vLLM's KV cache events by default and expose the ZMQ endpoint port (5557) on in-cluster Services so KV-cache-aware consumers running in the same cluster (in particular the llm-d Router / EPP that KAITO's GAIE integration wires up, but also any custom llm-d Router deployment or external cache coordinator) can subscribe.

The publisher-side cost is negligible even when nothing is subscribed (one background ZMQ thread per pod + a listening TCP socket on 5557), the socket is only reachable in-cluster (ClusterIP only, never on LoadBalancer), and users can still opt out by overriding --kv-events-config via --kaito-config-file.

What this PR does

1. Enable KV cache events on vLLM — The operator injects --kv-events-config='{"enable_kv_cache_events":true}' into the vLLM run command (in buildVLLMInferenceCommand) when the workspace runtime is vLLM and the user hasn't already set --kv-events-config themselves. This routes through vLLM's normal --kv-events-config CLI parser, so users can still override it via --kaito-config-file. Doing it operator-side means no image rebuild is needed to toggle the feature.

2. Advertise the ZMQ port on the vLLM pod and Service — Add port 5557 (kv-events) as a container port on vLLM inference pods and expose it as a Service port so in-cluster subscribers can connect.

3. Don't publish the ZMQ port externally — The KV events ZMQ stream is unauthenticated / unencrypted. The Service port is only added when the Service type is ClusterIP; if the workspace opts into a LoadBalancer Service (kaito.sh/enablelb: "True"), the kv-events port is intentionally not added. Users who genuinely need external access should create their own Service + NetworkPolicy.

What events are actually published

vLLM emits a single KVEventBatch message stream over ZMQ (defined in vllm/distributed/kv_events.py). Each batch contains one or more of the following per-block lifecycle events:

Event	When emitted	Notable payload
BlockStored	A KV cache block becomes resident (prefill fills it, or a decode step allocates a new block). One event per contiguous run of newly-populated blocks.	block_hashes[], parent_block_hash, token_ids[], block_size, medium (GPU/CPU/STORAGE), lora_name, extra_keys[] (multimodal ids, cache_salt, prompt embedding hashes — required for prefix-aware routing)
BlockRemoved	A block is evicted (LRU pressure, request finish, admin flush).	block_hashes[], medium, group_idx
AllBlocksCleared	Engine-level cache reset (e.g. reload/reset endpoint).	(no payload)

Every batch carries a top-level ts (timestamp) and, in data-parallel deployments, a data_parallel_rank so consumers can attribute events to the correct DP rank. Consumers see them as msgspec-encoded, tagged union messages over ZMQ PUB/SUB (topic prefix optional). vLLM also opens an optional ROUTER replay endpoint so subscribers can request missed batches by starting sequence number — that endpoint is not exposed by this PR.

What llm-d Router / EPP config is needed to actually consume them

Just wiring the vLLM producer up is not enough — the llm-d Router (formerly llm-d-inference-scheduler) EPP has to be configured with a plugin that subscribes to tcp://<pod>:5557 and turns those events into routing signals. Concretely, the EPP's EndpointPickerConfig needs to enable a KV-events-consuming producer and a prefix-cache scorer that references it. The canonical setup (from the upstream architecture doc and the precise-prefix-cache-producer plugin README) looks like:

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: precise-prefix-cache-producer            # ← this plugin owns the ZMQ subscriber lifecycle
  parameters:
    tokenProcessorConfig:
      blockSize: 16
    indexerConfig:
      kvBlockIndexConfig:
        maxPrefixBlocksToMatch: 256
    kvEventsConfig:                              # ← controls the ZMQ subscriber pool
      engineType: vllm                           # default; set to "sglang" for SGLang
- type: prefix-cache-scorer
  parameters:
    prefixMatchInfoProducerName: precise-prefix-cache-producer   # ← must reference the producer above
- type: decode-filter
- type: max-score-picker
- type: single-profile-handler
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: decode-filter
  - pluginRef: max-score-picker
  - pluginRef: prefix-cache-scorer
    weight: 50

Key points:

• precise-prefix-cache-producer is the plugin that actually spawns a per-pod ZMQ SUB and ingests BlockStored / BlockRemoved / AllBlocksCleared. Its EndpointExtractor hook manages the subscriber lifecycle on pod add/delete against the InferencePool.
• The default approx-prefix-cache-producer (which is what the EPP auto-injects when nothing is configured) is heuristic-only and does not consume KV events. You have to explicitly configure precise-prefix-cache-producer (or another KV-events-consuming producer) for the KV events KAITO is now emitting to be used.
• prefix-cache-scorer must set prefixMatchInfoProducerName to the producer's name; without it the scorer falls back to the approx producer and the KV events are effectively ignored.
• The producer requires a TokenizedPrompt on each request, which is provided upstream by the default token-producer — no extra config needed for that.
• Config is passed to the EPP as --config-file <path> or --config-text '<inline yaml>'.
• The EndpointPickerConfig API (apiVersion: llm-d.ai/v1alpha1) and the precise-prefix-cache-producer / prefix-cache-scorer plugins are provided by the llm-d-inference-scheduler EPP, not upstream gateway-api-inference-extension. Upstream GAIE only ships the lightweight EPP (lwepp) which does not consume KV events. If you're bringing your own EPP, make sure it's the llm-d one (or another KV-events-aware fork).
• Cross-engine caveat (from the producer README): vLLM emits extra_keys (including cache_salt) in its KV events, so salted requests are routed precisely. SGLang currently does not emit extra_keys, so salted requests fall back to precise-cache misses until SGLang catches up.

On the KAITO side, the InferenceSet-managed InferencePool and the KAITO GAIE integration already point the Router at the InferencePool fronting the vLLM workspace pods, so once the EPP config above is applied the subscribers connect to the kv-events port this PR is exposing. Consumers deployed independently of the KAITO MRI controller can connect the same way — the producer side is on for every vLLM workspace, not gated on any feature flag.

Changes

File	Change
pkg/utils/consts/consts.go	Add PortKVCacheEvents = 5557 constant
pkg/model/interface.go	In buildVLLMInferenceCommand, inject --kv-events-config='{"enable_kv_cache_events":true}' when not already set by the user
pkg/workspace/inference/preset_inferences.go	Name the existing HTTP container port http; add a separate kv-events container port (5557) for vLLM workspaces
pkg/workspace/manifests/manifests.go	In GenerateServiceManifest, add the kv-events Service port for vLLM workspaces when the Service type is ClusterIP (LoadBalancer excluded to avoid publishing the unauth ZMQ stream)

Unit tests added / updated:

• pkg/model/interface_test.go
  • TestGetInferenceCommandVLLMKVCacheEventsDefault — asserts default injection and that a user override wins.
• pkg/workspace/manifests/manifests_test.go
  • TestGenerateServiceManifest_KVEventsPort — vLLM/ClusterIP exposes the port; vLLM/LoadBalancer does not; non-vLLM/ClusterIP does not.
• pkg/workspace/inference/preset_inferences_test.go
  • TestGeneratePresetInference — asserts expectedParams["kv-events-config"] in the generated vLLM command.

Why (feature value)

KV cache events provide observability into vLLM's KV cache lifecycle (BlockStored, BlockRemoved, AllBlocksCleared), which powers:

• KV-cache-aware routing / prefix affinity in the llm-d Router (precise-prefix-cache-producer + prefix-cache-scorer)
• Monitoring cache utilization and eviction patterns
• External cache management and coordination

The events are published via ZMQ as part of vLLM's KVEventsConfig. Only enable_kv_cache_events needs to be explicitly set — all other parameters (publisher, endpoint, hwm, etc.) use sensible defaults.

Testing

• go test ./pkg/model/... ./pkg/workspace/manifests/... ./pkg/workspace/inference/... passes locally
• Verified --kv-events-config is present in the generated vLLM command for vLLM workspaces and absent for HuggingFace-transformers workspaces
• Verified the ClusterIP Service exposes the kv-events port for vLLM workspaces and does not expose it for LoadBalancer Services or non-vLLM runtimes
• End-to-end verified on a running phi-4 workspace: pod has containerPort/kv-events=5557, Service has kv-events port, vLLM log shows Starting ZMQ publisher thread, live ZMQ SUB receives BlockStored/BlockRemoved events — see demo/llm/kv-events.md for the full walkthrough and consumer-side analysis (GAIE + llm-d).

[Copilot]

Pull request overview

Enables vLLM KV-cache lifecycle event publishing by default and updates the Kubernetes-generated pod/service manifests to expose the KV-events ZMQ port (5557) so clients can subscribe to those events.

Changes:

• Enable kv_events_config.enable_kv_cache_events=True in the vLLM preset default args.
• Add a new PortKVCacheEvents = 5557 constant and wire it into generated container/service ports.
• Name the existing inference port as http and add a new kv-events port to the container/service definitions.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
presets/workspace/inference/vllm/inference_api.py	Enables KV cache events by default via vLLM default args (kv_events_config).
pkg/workspace/manifests/manifests.go	Exposes kv-events (5557/TCP) on the generated Workspace Service.
pkg/workspace/inference/preset_inferences.go	Adds http and kv-events container ports to the inference container spec.
pkg/utils/consts/consts.go	Introduces PortKVCacheEvents constant (5557).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kaito-pr-agent]

Failed to generate code suggestions for PR

[github-actions]

This pull request has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pkg/workspace/manifests/manifests_test.go:333

• The comment suggests this test can be made flaky by other packages’ tests mutating featuregates.FeatureGates under parallel go test runs. Go runs each package’s tests in a separate process, so other packages can’t affect this package’s globals; this rationale is misleading. Reword the comment to reflect that you’re pinning global mutable feature-gate state for determinism within this test/process.

	// Deterministically pin the vLLM feature gate for this test. Other packages'
	// tests mutate featuregates.FeatureGates (sometimes flipping FeatureFlagVLLM
	// to false or replacing the whole map) without restoring it, which would
	// otherwise make kaitov1beta1.GetWorkspaceRuntimeName here flaky under
	// parallel `go test` runs.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pkg/workspace/manifests/manifests_test.go:333

• The comment explains flakiness as coming from “other packages' tests” mutating featuregates.FeatureGates. go test runs each package’s tests in its own process, so cross-package mutations can’t affect this test; if there’s flakiness risk, it’s from other tests in the same package/test binary mutating the global feature-gate map. Rewording avoids misleading future maintainers about how/why this pinning is needed.

	// Deterministically pin the vLLM feature gate for this test. Other packages'
	// tests mutate featuregates.FeatureGates (sometimes flipping FeatureFlagVLLM
	// to false or replacing the whole map) without restoring it, which would
	// otherwise make kaitov1beta1.GetWorkspaceRuntimeName here flaky under
	// parallel `go test` runs.