feat: migrate to llm-d-router-gateway chart v0.9.0 and enable KV cache-aware routing

[andyzhangx]

What type of PR is this?

/kind feature

What this PR does / why we need it

Migrates the InferencePool Helm chart dependency from the Gateway API Inference Extension (GWIE) inferencepool chart to the llm-d-router-gateway chart at v0.9.0, and enables KV cache-aware routing in the default EPP EndpointPickerConfig.

Why migrate?

1. Bool flags rendering bug — The GWIE chart (v1.3.1) renders boolean flags as two separate args (--key "value"), which breaks Go pflag parsing (see llm-d/llm-d-router#924, kubernetes-sigs/gateway-api-inference-extension#2871). The llm-d-router chart already has this fix.
2. Better alignment — KAITO already uses the llm-d EPP image; using the llm-d chart directly removes the need for image override hacks and ensures chart+image version consistency.
3. Advanced features — The llm-d-router chart supports proxy mode, tokenizer sidecar, latency predictor, and other features that KAITO can leverage in future PRs.

KV cache-aware routing

With #1890 enabling KV cache events on vLLM pods by default, this PR adds the kv-cache-utilization-scorer plugin to the default P/D EndpointPickerConfig. The scorer reads each endpoint's KV cache utilization from vLLM metrics (/metrics) and computes score = 1 - kvCacheUsagePercent, spreading traffic away from pods with high KV cache pressure.

• Plugin wired into both prefill and decode scheduling profiles with weight 5 (half of load-aware-scorer's weight 10)
• Queue depth (load-aware-scorer) remains the primary routing signal; KV cache pressure acts as a secondary signal
• No additional sidecar or ZMQ subscription is required — the scorer consumes the same metrics endpoint EPP already scrapes

Why kv-cache-utilization-scorer and not prefix-cache-scorer with precise-prefix-cache-producer?

The llm-d Router provides two levels of prefix-cache-aware routing:

	Approximate (current)	Precise (future)
Producer	approx-prefix-cache-producer	precise-prefix-cache-producer + token-producer
How it works	EPP approximates tokens using character-to-token ratios, maintains an in-memory LRU index of prefix hashes	Subscribes to vLLM KV cache events via ZMQ, maintains a globally consistent index of exact token blocks per pod
Accuracy	Heuristic (character-based)	100% (token-based)
Dependencies	None beyond EPP	Tokenizer sidecar (vLLM render endpoint), ZMQ connectivity to each vLLM pod, KV-Cache Indexer
Resource overhead	Negligible	~500m CPU / 1Gi memory per MRI (for tokenizer sidecar)

This PR stays with the approximate approach because:

1. No tokenizer sidecar — The precise-prefix-cache-producer requires a token-producer plugin that sends prompts to vLLM's /v1/completions/render endpoint (typically a vllm launch render sidecar in the EPP pod). KAITO does not currently deploy this sidecar, and adding one would cost ~500m CPU / 1Gi memory per MRI with no benefit for the approximate pipeline.

2. KV-Cache Indexer not configured — While #1890 enables the ZMQ publisher on vLLM pods (port 5557), the EPP side's KV-Cache Indexer is not yet configured to subscribe to these events. The precise-prefix-cache-producer depends on this indexer to maintain the global block-to-pod mapping.

3. kv-cache-utilization-scorer is complementary, not overlapping — It reads the kvCacheUsagePercent metric from vLLM's /metrics endpoint (which EPP already scrapes for load-aware-scorer), requiring zero additional infrastructure. It provides a different signal: how full is each pod's KV cache (distribution/load-balancing), vs. how much of this specific request's prefix is cached on each pod (affinity). Both are valuable.

4. The approximate producer is already in the config — approx-prefix-cache-producer + prefix-based-pd-decider handles prefix-aware P/D routing without any external dependencies. Adding a prefix-cache-scorer on top would partially overlap with the decider's routing decisions in the P/D context.

Next steps: Upgrade to precise prefix-cache routing (follow-up PRs)

Upgrading from the current approximate approach to precise prefix-cache routing will bring significant improvements, especially under high load and in P/D disaggregation scenarios:

What precise routing improves

Aspect	Approximate (current)	Precise (future)
Cache state awareness	EPP guesses — "I sent prefix X to pod-2 last time, so pod-2 should still have it"	EPP knows — vLLM sends real-time BlockStored / BlockRemoved / AllBlocksCleared events via ZMQ, EPP maintains a globally consistent index
Eviction handling	❌ EPP's LRU assumption breaks when pods evict blocks under memory pressure → stale routing → unnecessary prefill recomputation	✅ EPP receives BlockRemoved events immediately → never routes to a pod that already evicted the prefix
Tokenization accuracy	Character-to-token ratio approximation, hash block boundaries may not align with real tokens → cache miss even when cache exists	Exact Token IDs from vLLM render endpoint → if cache is there, it will hit
P/D transfer optimization	Coarse-grained prefill/decode split only	Precise knowledge of which blocks are on which pod → optimal KV transfer decisions between prefill and decode pods
Cache hit rate	Good for homogeneous workloads with low eviction	Significantly higher under high load, heterogeneous prompts, or frequent cache eviction

Concrete example

A user sends multiple prompts sharing a long system prompt (e.g., 4K tokens of instructions):

• Approximate: EPP hashes the system prompt characters → routes to pod-3 (where it sent a similar prefix before). But pod-3 evicted those blocks 30 seconds ago due to memory pressure. Result: full prefill recomputation (~2s latency penalty).
• Precise: EPP's KV-Cache Indexer received BlockRemoved from pod-3, knows pod-1 still has 90% of the blocks cached. Routes to pod-1. Result: cache hit, near-zero prefill latency.

Implementation roadmap

Phase	PR	What	Prerequisite
Phase 1	Tokenizer sidecar	Add optional vllm launch render sidecar to EPP deployment, gated by feature flag or MRI spec field. Enables token-producer plugin for exact tokenization.	llm-d-router chart supports sidecar containers
Phase 2	KV-Cache Indexer	Configure EPP to subscribe to vLLM KV cache events on ZMQ port 5557 (already exposed by #1890). Maintains real-time block→pod mapping.	Phase 1 (token-producer needed for block hash computation)
Phase 3	Precise prefix-cache routing	Add precise-prefix-cache-producer + update prefix-cache-scorer with prefixMatchInfoProducerName: precise-prefix-cache-producer. Achieves 100% accurate prefix-cache-aware routing.	Phase 1 + Phase 2
Phase 4	Speculative indexing	Enable precise-prefix-cache-producer's speculative indexing to proactively update the index after routing decisions, closing the gap between routing and KV event arrival.	Phase 3

Trade-offs

• Cost: Tokenizer sidecar adds ~500m CPU / 1Gi memory per MRI; ZMQ connections add minor network overhead
• Complexity: More moving parts (sidecar lifecycle, ZMQ connectivity, indexer consistency)
• When to upgrade: Production deployments with high QPS, shared long prefixes (RAG, multi-turn chat), or latency-sensitive workloads where prefill recomputation is the bottleneck

Changes

Component	Before	After
Chart URL	oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool	oci://ghcr.io/llm-d/charts/llm-d-router-gateway
Chart version	v1.3.1	v0.9.0
EPP image	mcr.microsoft.com/oss/v2/llm-d/llm-d-inference-scheduler:v0.8.0	mcr.microsoft.com/oss/v2/llm-d/llm-d-router-endpoint-picker:v0.9.1
Values schema	inferenceExtension.image.{hub,name}	router.epp.image.{registry,repository}
Values schema	inferencePool.{targetPorts,modelServers}	router.modelServers.{targetPorts,matchLabels}
EPP config	load-aware-scorer only	load-aware-scorer + kv-cache-utilization-scorer

Files changed

File	Change
pkg/utils/consts/consts.go	Update chart URL, version (v0.9.0), and EPP image constants
pkg/workspace/manifests/manifests.go	Update Helm values to match llm-d-router schema
pkg/controllers/multiroleinference/controller.go	Update schema + add kv-cache-utilization-scorer to default EndpointPickerConfig
pkg/workspace/manifests/manifests_test.go	Update expected values in tests
test/e2e/preset_vllm_test.go	Add e2e validation for kv-cache-utilization-scorer in EPP ConfigMap and pod logs

Which issue(s) this PR fixes

Resolves llm-d/llm-d-router#924 (for KAITO users)

Related PRs

• feat: enable KV cache events on vLLM inference pods by default #1890 (enable KV cache events on vLLM pods by default)
• fix: helm chart flags rendering for boolean values kubernetes-sigs/gateway-api-inference-extension#2871 (upstream bool flag fix)
• migrate: sigs.k8s.io/gateway-api-inference-extension @ dc908434 llm-d/llm-d-inference-payload-processor#17 (llm-d port of the fix)

[Copilot]

Pull request overview

Migrates KAITO’s InferencePool Helm chart integration from the Gateway API Inference Extension (GWIE) inferencepool chart to the llm-d-router-gateway chart, updating constants and Helm values to the new schema while keeping the existing InferencePool/EPP behavior aligned with llm-d-router.

Changes:

• Update chart source (OCI URL) and version constants, along with the EPP image constants, to llm-d-router-gateway v0.9.1.
• Update generated Helm values from the old inferenceExtension/inferencePool schema to the new router.epp/router.modelServers schema.
• Update manifest unit tests to assert the new values structure.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
pkg/utils/consts/consts.go	Switch InferencePool chart URL/version and EPP image constants to llm-d-router-gateway + new image fields (registry/repository).
pkg/workspace/manifests/manifests.go	Update generated Flux OCIRepository comment + HelmRelease values to router.epp.image and router.modelServers shape.
pkg/controllers/multiroleinference/controller.go	Update MRI controller’s HelmRelease values to the new llm-d-router schema (router.epp, router.modelServers).
pkg/workspace/manifests/manifests_test.go	Adjust expected Helm values in tests to match the new router.* schema and targetPorts nesting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/controllers/multiroleinference/controller.go:666

• These EPP resource defaults are duplicated with the InferenceSet manifests path (pkg/workspace/manifests/manifests.go). Keeping two separate copies increases the chance they diverge over time and makes sizing changes harder to reason about. Consider consolidating them into a shared helper/const so both controllers use the same source of truth.

	eppValues["resources"] = map[string]any{
		"requests": map[string]string{
			"cpu":    "1",
			"memory": "2Gi",
		},
		"limits": map[string]string{
			"memory": "16Gi",
		},
	}

pkg/workspace/manifests/manifests.go:451

• The EPP resource requests/limits are hard-coded here and the same defaults are also duplicated in the MultiRoleInference controller. This duplication makes it easy for the two code paths to drift (InferenceSet vs MRI) and makes future tuning harder. Consider factoring these defaults into a shared helper/const (e.g., in pkg/utils/consts or a manifests helper) and referencing it from both places, or wiring them from API/config if they should be user-tunable.

				"resources": map[string]any{
					"requests": map[string]string{
						"cpu":    "1",
						"memory": "2Gi",
					},
					"limits": map[string]string{
						"memory": "16Gi",
					},
				},

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

pkg/controllers/multiroleinference/controller.go:522

• The comment says kv-cache-utilization-scorer leverages vLLM KV cache events, but this scorer is intended to use vLLM metrics (KV cache utilization) as its signal. Keeping the comment accurate matters because KV events are a separate integration path (ZMQ/indexer) and can mislead future readers.

// kv-cache-utilization-scorer leverages vLLM KV cache events (enabled by default on vLLM pods)
// to score endpoints by available KV cache capacity, spreading traffic away from high-pressure pods.

pkg/controllers/multiroleinference/controller.go:664

• This change drops the explicit secure-serving=false flag for EPP. The repo docs indicate EPP defaults to --secure-serving=true (self-signed cert), which requires an Istio DestinationRule TLS bypass; removing the flag may unintentionally revert to secure serving and change Gateway→EPP connectivity expectations. If the goal is still to disable secure serving, set the flag here explicitly.

	eppValues["flags"] = map[string]string{
		"model-server-metrics-port": fmt.Sprintf("%d", consts.PortInferenceServer),
	}

test/e2e/preset_vllm_test.go:1482

• The fallback path scans all ConfigMaps in the namespace and only checks whether any value contains the substring kv-cache-utilization-scorer. This can produce false positives (unrelated ConfigMaps) and makes the test more expensive/flaky in busy namespaces. Restrict the fallback scan to ConfigMaps whose name is clearly associated with this InferencePool (e.g., prefixed by poolName).

				for _, cm := range cmList.Items {
					for _, v := range cm.Data {
						if strings.Contains(v, "kv-cache-utilization-scorer") {
							GinkgoWriter.Printf("ConfigMap %s contains kv-cache-utilization-scorer\n", cm.Name)
							found = true

pkg/workspace/manifests/manifests.go:447

• The EPP resource requests/limits are hard-coded here and the exact same values are also hard-coded in the MultiRoleInference controller. Duplicating these defaults in multiple places risks configuration drift when one side is updated. Consider centralizing these defaults (e.g., helper in a shared package or consts) so both code paths stay in sync.

				"resources": map[string]any{
					"requests": map[string]string{
						"cpu":    "1",
						"memory": "2Gi",
					},

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

pkg/controllers/multiroleinference/controller.go:522

• The comment says kv-cache-utilization-scorer leverages vLLM KV cache events, but this scorer is described elsewhere in this PR as using vLLM /metrics KV cache utilization. Updating the comment will avoid misleading readers about ZMQ/KV-events dependencies.

// kv-cache-utilization-scorer leverages vLLM KV cache events (enabled by default on vLLM pods)
// to score endpoints by available KV cache capacity, spreading traffic away from high-pressure pods.

test/e2e/preset_vllm_test.go:1460

• Get() errors for candidate ConfigMaps are currently ignored unconditionally. This can mask real failures (e.g., RBAC/connection issues) and makes the test harder to debug. Only ignore NotFound and surface other errors.

				cm, err := coreClient.CoreV1().ConfigMaps(mriObj.Namespace).Get(ctx, name, metav1.GetOptions{})
				if err != nil {
					continue
				}

test/e2e/preset_vllm_test.go:1479

• The fallback path scans every ConfigMap in the namespace, which can be slow and can match unrelated resources (especially if previous e2e runs left artifacts). Narrow the scan to ConfigMaps whose names include this MRI's pool name.

				for _, cm := range cmList.Items {
					for _, v := range cm.Data {