Routing agent traffic is really three decisions

A chat turn is one LLM call. An agent is hundreds — a reasoning loop that plans, calls a tool, reads the result, re-plans, and only sometimes stops. The cost and latency you signed up for in a demo get multiplied by that loop, and most of those calls are easy — a tool-argument fill, a yes/no gate, a short summary — that never needed a frontier model. So the question isn't "which model is best." It's "which model should answer this specific call, and how do I govern a flood of them?" That's not one decision; it's three, on different signals.

This post wires up those three decisions on AKS as a single OpenAI-compatible endpoint: a trivial agent step lands on a cheap self-hosted model (KAITO + vLLM, placed by GPU state) while a hard step escalates to a frontier model on Azure OpenAI — every call authenticated, metered, and traced. We lean on managed Azure throughout (KAITO for serving, Azure OpenAI for the frontier path, Azure Managed Prometheus and Grafana for observability); the two layers without a drop-in managed service — the semantic router and the AI gateway — run as open source on the cluster.

The three decisions​

"Route this request" hides three separate questions, each reading a different signal, each with a different owner. For an agent firing calls in a loop, the first two fire on every call and the third on every weak-path call:

1. Which model should answer this call? A multi-step reasoning hop deserves a frontier model; "extract the date from this string" does not. Answering means reading the request content and predicting answer quality. This is semantic routing, and we'll use RouteLLM.
2. How do I expose, secure, govern, and forward this traffic? Provider abstraction, auth, per-agent budgets, rate limits, guardrails, observability. None of that cares about prompt meaning — it's a policy job for a data plane. We'll use agentgateway, an OpenAI-compatible, agent-native proxy, in the cluster.
3. Which replica of the chosen model gets this call? Across a fleet of GPU pods the right target depends on KV-cache occupancy, queue depth, and which LoRA adapters are already resident — live GPU state. That's the Gateway API Inference Extension and its Endpoint Picker.

Content → policy → GPU state. That's the through-line, and the discipline is refusing to let any layer answer a question that belongs to another: the router never thinks about auth, the gateway never thinks about prompt difficulty, the Endpoint Picker never thinks about which model — only which replica.

The architecture on AKS​

Most slots are managed Azure; where an open-source piece has a managed alternative, it's called out in the layer that earns it:

With the map in hand, the rest of this post builds it — bottom-up, serving first and the semantic decision last, because each layer needs the address of the one beneath it. First, the cluster everything runs on.

Prerequisites​

• An Azure subscription with GPU quota for the VM family you'll serve on (this guide uses a single Standard_NC* GPU node pool, provisioned for you by KAITO). Quota isn't capacity — confirm the SKU is actually allocatable in your region before you rely on it.
• A recent Azure CLI and kubectl.
• An Azure OpenAI (or AI Foundry) resource with a current frontier deployment for the strong path. We use gpt-5.1; gpt-4o is deprecating in many regions and may not be newly deployable.

Create a cluster with the serving and monitoring add-ons enabled. KAITO provisions and manages GPU node pools on demand, so you don't pre-create them:

az aks create \
  --resource-group $RG \
  --name $CLUSTER \
  --node-count 2 \
  --enable-ai-toolchain-operator \
  --enable-oidc-issuer \
  --enable-azure-monitor-metrics \
  --generate-ssh-keys

az aks get-credentials --resource-group $RG --name $CLUSTER

That's the whole cluster — no gateway add-on, no extra CRDs, no Istio to bootstrap. agentgateway is the front door (more on that in Layer 2). With the cluster up, start at the bottom: serving the weak model.

Layer 3 — Serving, and the load-balancing trap​

Start with the failure mode, because it's the whole reason this layer is more than a Service — and agents make it worse.

Round-robin assumes requests are interchangeable. LLM requests are anything but: a 100k-token context-stuffed agent step and a 200-token tool-argument fill differ by three orders of magnitude in cost. An agent fleet mixes both constantly. Spray them uniformly across identical pods and round-robin will cheerfully queue a cheap completion behind a giant prefill on a saturated pod while an idle pod sits next to it. Your throughput looks fine and your p99 — the thing an agent loop feels on every iteration — falls off a cliff.

The fix isn't more pods — it's placing each request on the pod best able to take it right now, by reading live KV-cache occupancy and queue depth. That capability is what the Endpoint Picker is. Everything else in this layer exists to feed it.

3.1 The pods to place: KAITO​

The Endpoint Picker needs a homogeneous pool of model servers exporting the right signals, and KAITO produces exactly that: declare a Workspace, point it at a preset, and KAITO provisions the GPU SKU, runs vLLM, and exposes an OpenAI-compatible Service. Because the engine is vLLM, every pod emits the metrics the Endpoint Picker routes on (vllm:num_requests_waiting, vllm:kv_cache_usage_perc). Apply kaito-workspace.yaml — it pins instanceType: Standard_NC24ads_A100_v4, count: 2, and preset: phi-4-mini-instruct:

kubectl create namespace llm
kubectl apply -f kaito-workspace.yaml
kubectl get workspace workspace-phi-4-mini -n llm -w   # Ready can take ~10–20 min

KAITO provisions the GPU node, pulls the model, and starts vLLM; the workspace flips to Ready when inference is up:

NAME                   INSTANCE                   RESOURCEREADY   INFERENCEREADY   STATE
workspace-phi-4-mini   Standard_NC24ads_A100_v4   True            True             Ready

KAITO publishes a Service named workspace-phi-4-mini on port 80 and labels its pods kaito.sh/workspace: workspace-phi-4-mini — that label is how the next step selects the pool. One detail that will bite you later: the Service maps port 80 to the vLLM container's port 5000, and the Endpoint Picker routes to pod IPs directly, bypassing the Service. So the pool below targets 5000, not 80.

3.2 The placement logic: the Inference Extension​

Pool membership — which pods are the model server — is a different concern from how a workload is weighted against that pool, so the Inference Extension splits them across two objects:

• An InferencePool — the set of model-server pods plus a reference to the Endpoint Picker (EPP), the extension that an ext-proc-capable proxy calls to choose an endpoint per request. In this design that proxy is agentgateway (Layer 2); the InferencePool and EPP don't care who calls them.
• An InferenceObjective — the per-workload treatment: an integer priority against a pool. It replaces the older InferenceModel object: where InferenceModel carried a modelName and a criticality enum, InferenceObjective keeps only an integer priority and takes the served-model name from the request instead.

Install the CRDs and the Endpoint Picker (the project ships a Helm chart), pointing it at the KAITO pods. The EPP must serve plaintext gRPC to match agentgateway's ext-proc client — set --secure-serving=false, or the call to it resets before headers:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.0/manifests.yaml

helm install phi-epp \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool \
  --version v1.0.0 -n llm \
  --set inferencePool.modelServers.matchLabels."kaito\.sh/workspace"=workspace-phi-4-mini \
  --set provider.name=none \
  --set inferenceExtension.extraArgs="{--secure-serving=false}"

The Helm chart creates its own InferencePool named after the release, so you don't strictly need to apply one. If you do, inference-pool.yaml has the full InferencePool + InferenceObjective — with two fields that trip people up. The pool's selector needs a structured matchLabels, and its targetPorts must be the vLLM container port 5000 (the pool routes to pod IPs, bypassing the Service's port 80):

spec:
  selector:
    matchLabels: { kaito.sh/workspace: workspace-phi-4-mini }
  targetPorts:
    - number: 5000          # the vLLM container port, NOT the Service's 80

The other trap is the API group: InferencePool lives in inference.networking.k8s.io, but InferenceObjective lives in inference.networking.x-k8s.io — different groups, easy to miss.

The Endpoint Picker is now running and watching the pool, but nothing calls it yet. It needs an ext-proc-capable proxy out front to consult it per request — and that proxy is the AI gateway in Layer 2, so we wire the call there rather than standing up a separate Gateway API gateway just to reach it.

What this layer decided: only which replica, on GPU state. It has no idea what a "strong" model is, no budgets, no auth. Those are the next layer's job.

Layer 2 — The gateway, and why agents need it most​

The tempting shortcut is to let the router call backends directly. Resist it — and with agents the reason is sharper than with chat. An agent loop can spend an unbounded number of tokens before it decides to stop; without a budget and a rate limit somewhere on the path, a single misbehaving agent can run up a frontier bill or saturate your GPU fleet. A gateway is where that policy lives, written once, so nothing else has to reinvent it.

agentgateway is an open-source, OpenAI-compatible AI gateway built for exactly this traffic: provider abstraction, auth, token-based rate limiting, model-cost tracking, guardrails, and OpenTelemetry across backends — and, because it's agent-native, first-class MCP tool-serving and agent-to-agent routing when you grow into them. Here it does three jobs: hide the strong (Azure OpenAI) and weak (in-cluster) backends behind one endpoint, enforce policy on every call, and — because it speaks ext-proc itself — call the Endpoint Picker directly so the weak path gets GPU-state placement. That third job is what lets us drop the Gateway API gateway entirely: one data plane instead of two, and no dependency on the managed gateway's inference-extension support, which is still preview-gated on AKS today.

The outer shape is binds → listeners → routes → backends. The one thing to internalize: an AI backend is ai: { name, provider: { <provider>: {...} } } — the provider nests under provider:, name is required, and the Azure provider needs resourceName and resourceType (its model is your Azure deployment name). Auth attaches as a route policy whose key the Azure provider sends as the api-key header (a static key keeps the example self-contained; for production on AKS, prefer Azure workload identity, which agentgateway supports natively for the Azure provider):

# the strong route's backend (excerpt)
backends:
  - ai:
      name: aoai-strong
      provider:
        azure:
          resourceName: "<your-aoai-resource>"
          resourceType: openAI
          model: gpt-5.1
          apiVersion: "2024-10-21"

The strong route is a single ai backend pointed at Azure OpenAI. The weak route is where the front-door magic lives: a service backend pointed at the KAITO pods, with an inferenceRouting policy that hands each request to the Endpoint Picker over ext-proc and forwards to the pod it returns. Three details earned during validation, because the loader is strict about each:

# top-level: declare the model-server pool the picker selects from
services:
  - name: phi-weak
    namespace: llm
    hostname: workspace-phi-4-mini.llm.svc.cluster.local
    vips: []                       # required, even when empty
    ports: { 5000: 5000 }          # the vLLM container port

# ...the weak route's backend:
backends:
  - service:
      name: llm/workspace-phi-4-mini.llm.svc.cluster.local   # namespace/HOSTNAME, not the service name
      port: 5000
    policies:
      inferenceRouting:            # a BACKEND policy, not a route policy
        endpointPicker: { host: phi-epp-epp.llm.svc.cluster.local:9002 }
        destinationMode: passthrough   # trust the picker's choice

The full two-route config — including the strong ai backend and both routes' policies — is in agentgateway-config.yaml. The credential never leaves the gateway, and on the weak route agentgateway itself is the thing that consults the picker.

The governance that makes this safe for agents lives there too: the strong route carries a token rate limit (a localRateLimit route policy) — the backstop against an agent loop that won't stop spending. agentgateway can also price every call in real dollars through a top-level config.modelCatalog, but its schema is nested and provider-scoped — the obvious flat form didn't load for us — so wire it from the agentgateway docs for your release rather than copying a snippet.

Smoke-test both paths before the router exists:

# strong → Azure OpenAI
curl -s http://agentgateway.llm.svc.cluster.local:4000/strong/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.1","messages":[{"role":"user","content":"Say no in one word."}]}'

# weak → KAITO via the Endpoint Picker
# (same shape RouteLLM sends: it appends /v1/chat/completions to api_base)
curl -s http://agentgateway.llm.svc.cluster.local:4000/weak/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"phi-4-mini-instruct","messages":[{"role":"user","content":"Say hi in one word."}]}'

Both return an OpenAI-shaped completion — the strong call comes back stamped with the resolved Azure deployment, the weak call from the in-cluster model, and neither caller had to know where the model lived:

// strong → Azure OpenAI
{ "model": "gpt-5.1-2025-11-13",
  "choices": [{ "message": { "role": "assistant", "content": "No." } }],
  "usage": { "total_tokens": 29 } }

// weak → KAITO / vLLM
{ "model": "phi-4-mini-instruct",
  "choices": [{ "message": { "role": "assistant", "content": "Hello." } }] }

What this layer decided: how to authenticate, meter, guardrail, and forward — on policy, never on content. It dispatches whatever model name it's handed. Choosing that name is Layer 1's job.

Layer 1 — The semantic decision, and the threshold that is the whole ballgame​

Here's the only layer that reads the prompt — and the one decision that actually moves the bill. For an agent firing hundreds of calls per task, it's also the layer with the most leverage: every easy call it keeps off the frontier model is multiplied by the loop.

RouteLLM is a set of routers trained on human-preference data to predict, per prompt, whether a cheap model can match a strong model's answer. On the paper's own model pair, the featured mf (matrix-factorization) router reaches ~95% of GPT-4's quality on MT-Bench while making just ~26% of the GPT-4 calls — cutting cost by up to ~85% versus sending everything to GPT-4, by escalating only the genuinely hard prompts.

The composition is the elegant part: RouteLLM's strong and weak upstreams are just agentgateway. It makes the semantic call, picks a model, and forwards into the governed data plane — so the decision layer inherits all of Layer 2's auth, budgets, and tracing without re-implementing a line of it.

One wiring detail the reviewer in you should catch: agentgateway routes by path (/strong, /weak), but RouteLLM hands its choice to litellm as a model name. So point each model at the matching agentgateway path with a per-model api_base rather than a single base URL:

# inside the RouteLLM container — each model maps to its agentgateway route
python -m routellm.openai_server --routers mf \
  --strong-model openai/gpt-5.1 \
  --weak-model   openai/phi-4-mini-instruct \
  --config routellm-config.yaml   # sets api_base per model → /strong and /weak

One more gotcha from running this against current models: RouteLLM sends presence_penalty / frequency_penalty by default, and litellm (which RouteLLM uses to reach each provider) raises UnsupportedParamsError when the target model doesn't accept them — newer frontier models like the gpt-5.x family don't. Set litellm.drop_params=True so litellm silently drops the unsupported params instead of failing the call.

Clients call RouteLLM with a model string that encodes both the router and the cost threshold:

router-mf-0.11593
        │   └── threshold: the strong/weak cutoff — calibrate it
        └────── the matrix-factorization router

That threshold is the ballgame. Set it too low and you escalate everything and pay frontier prices for "hello"; set it too high and quality craters on the hard steps that needed the strong model. Don't guess it — calibrate against a representative sample so a target fraction of your traffic escalates:

python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5
# → 0.11593: the threshold that sends ~50% of calls to "strong"

Then close the loop with the real numbers. Run live agent traffic and read the actual strong/weak split off agentgateway's metrics — not RouteLLM's estimate — and, once you've wired the cost catalog, the dollars too; nudge the threshold until the cost/quality trade sits where you want it. The router proposes; your own traffic disposes.

One caveat the rate card can't see: prompt caching. A cached input token costs a fraction of an uncached one — Azure OpenAI discounts cache hits, and the weak path's vLLM prefix cache is the very thing the Endpoint Picker routes on. Both caches are per-model and per-prefix, so per-call routing works against them: bounce a session between models and each one's cache for that growing context goes cold, so the marginal cost of a strong call whose prefix is still warm is much lower than its sticker rate, and the savings from offloading it are smaller than the headline prices imply. This is another reason to measure billed cost rather than multiply tokens by a static rate — and, where caching dominates over difficulty, to route at session or task granularity instead of per call. Per-call routing on prompt difficulty is still the right default; just don't let a token-rate estimate convince you the split is cheaper or more expensive than your bill says.

Observability with managed Azure monitoring​

Layer 1 trades quality for cost, so this section closes one question: can you trust the routing numbers, and can you watch the same signals the Endpoint Picker routes on? The answer splits by source — GPU and latency come from vLLM; the strong/weak split (and, once you wire the cost catalog, dollars) comes from agentgateway — so a complete view scrapes both.

To get there, add two PodMonitors to the Managed Prometheus you enabled at cluster creation: one selecting the KAITO pods for vLLM's metrics on port 5000, one for agentgateway's on port 15020. Then link Azure Managed Grafana to the workspace. (One gotcha: the managed add-on uses Azure's CRDs under the azmonitoring.coreos.com API group, not upstream monitoring.coreos.com, so change a community manifest's apiVersion to azmonitoring.coreos.com/v1 or the add-on won't scrape it.)

The full path, one call​

# trivial step → router picks weak → agentgateway → Endpoint Picker → KAITO pod
curl -s http://routellm.llm.svc.cluster.local:6060/v1/chat/completions \
  -d '{"model":"router-mf-0.11593","messages":[{"role":"user","content":"What is 2+2?"}]}'

# hard step → router picks strong → agentgateway → Azure OpenAI
curl -s http://routellm.llm.svc.cluster.local:6060/v1/chat/completions \
  -d '{"model":"router-mf-0.11593",
       "messages":[{"role":"user","content":"Derive the EM algorithm and explain why each step cannot decrease the log-likelihood."}]}'

The model field in each response is the receipt — it tells you which way the router decided. Against the calibrated threshold 0.11593, "What is 2+2?" scores a strong-win-rate of 0.09 (below the line → weak) while the EM-algorithm prompt scores 0.29 (above → strong), so they come back from different models:

// "What is 2+2?"             → "model": "phi-4-mini-instruct"   (weak, a fraction of a cent)
// "Derive the EM algorithm…" → "model": "gpt-5.1-2025-11-13"    (strong, frontier pricing)

Both hit the same endpoint with the same model string. Three decisions — content, policy, GPU state — resolved in one round trip, each by the layer that owns it. Now point your agent framework's base URL at that endpoint and every call in the loop gets sorted the same way.

Scale it to your problem​

You rarely need all of it on day one. Because the layers don't bleed into each other, you can adopt exactly the slice your problem demands:

• One hosted model, a few agents, no self-hosting? Layer 2 only — agentgateway for auth, budgets, and guardrails over Azure OpenAI. There's nothing to route to and no GPUs to place on.
• Self-hosting at scale but one model class? KAITO + the Inference Extension give you managed serving and cache-aware placement. Skip semantic routing — there's only one model.
• A real cost problem from agents sending easy steps to expensive models? Add RouteLLM. It earns its keep the moment you have a meaningful price gap and a meaningful fraction of easy traffic — which describes essentially every agent in a loop.

Conclusion​

Routing agent traffic isn't one decision — it's three, each on its own signal: content picks the model (RouteLLM), policy governs the call (agentgateway), and GPU state picks the replica (the Inference Extension over KAITO-served vLLM). Keep those seams clean and you get a single OpenAI-compatible endpoint your agents point at, where every call is routed, metered, and traced by the layer that owns that concern — and where the easy calls an agent loop fires by the hundred quietly stay off the frontier bill.

From here, the obvious next moves all reinforce the same shape: autoscale the weak fleet on vllm:num_requests_waiting with the managed KEDA add-on so the GPU pool breathes with bursty agent traffic; lean into agentgateway's agent-native side by serving tools over MCP and routing agent-to-agent traffic through the same governed data plane; and add per-agent budgets at the gateway so spend stays attributable as more agents share the cluster.

The example manifests have everything here ready to adapt. For the components themselves, see KAITO on AKS, the Gateway API Inference Extension, agentgateway, RouteLLM (and its paper), and Managed Prometheus and Grafana for AKS.