Yes, you can use prompt caching with Auto-Routing. The two compound rather than cancel out. We measured it across five datasets, two of which report what the provider's cache actually did.

The results​

• Auto-routing does not break prompt caching. The two compound on every dataset we measured
• 37% to 69% cheaper than caching alone on a single model
• The real failure mode is the opposite one. Running a router with caching switched off is about 4x more expensive than caching one fixed model

How it was measured​

• Router arm: one model group, four tiers. SIMPLE to claude-haiku-4-5, MEDIUM to claude-sonnet-5, COMPLEX and REASONING to claude-opus-5
• Baseline arm: every request to one frontier model with prompt caching on, the strongest realistic baseline rather than a cold-priced strawman
• Cache behaviour: read from usage.cache_read_input_tokens and cache_creation_input_tokens on the gateway spend logs and agent traces; modelled on the other three legs
• Turn classification: one window function partitioned by session and model, labelling each turn as staying on a model, first visiting a tier, or returning to one already used

Why auto-routing doesn't break prompt caching​

99.3% of the time a session switches back to a model it used earlier, that model's cache is still warm. The router comes back long before the cache expires, so a switch is not an eviction.

We measured this on 4,684 real switch-backs from live LiteLLM gateway traffic.

We tried a background cache warmer. It wasn't worth it​

If switching really did strand the cache, a background refresher that replays a session's prefix would be the fix. We built the measurement for it first.

Only 4% of cache misses are preventable by a background cache warmer. The rest either happened while the cache was still alive, or after the model had been idle so long that keeping it warm costs more than the write it avoids.

Warming is worth roughly plus or minus two percent: a narrow optimization for long sessions with large stable prefixes, not the thing standing between a deployment and its savings.

See it on your own traffic​

The Auto-Router Benchmarks tab now reports prompt cache behaviour per router, from the provider's own usage payload:

• Hit rate, split by whether the turn stayed on a model, first visited a tier, or returned to one
• Expired-miss share, narrowing return misses to those whose tier went idle past the TTL
• Savable by warming, the share of all misses a refresher could prevent
• Warming cost and net estimate, in dollars
• Coverage, so a low hit rate caused by response logging being off does not read as a cold cache

GET /auto_router/benchmarks?start_date=2026-07-01&end_date=2026-07-31

Try it​

model_list:
  - model_name: claude-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE:    claude-haiku-4-5
          MEDIUM:    claude-sonnet-5
          COMPLEX:   claude-opus-5
          REASONING: claude-opus-5
        classifier_type: heuristic
      complexity_router_default_model: claude-sonnet-5

Every response carries x-litellm-model-name and x-litellm-response-cost, and the provider's cache token counts land in the spend logs. Full reference on the Auto Routing docs page.

Auto routing promises a smaller bill without a worse answer. We measured both halves against a baseline that sends every request to claude-opus-5: 8,619 graded prompts and cost simulations over 14,000 real conversations.

The results​

• 40.4% cheaper at 97.1% of frontier quality, on 220 prompts from six public benchmarks replayed through a live proxy
• 74.5% cheaper at 87.3% of frontier quality, on RouterArena's full 8,399-query set
• Around 65% cheaper on simulated real chat and developer traffic, where most requests are short

How it was measured​

• Router arm: one model group, four tiers. SIMPLE to claude-haiku-4-5, MEDIUM to claude-sonnet-5, COMPLEX and REASONING both to claude-opus-5, since Opus 5 already thinks by default. Heuristic classifier, no keyword rules, no adaptive sampling
• Baseline arm: the same prompts sent straight to claude-opus-5, which is what most teams do today when they point a workload at one frontier model
• Cost savings: what the router spent against what the baseline spent. On the live proxy, LiteLLM reports the cost of every request back in a response header; on the other legs we price the tokens each request actually used
• Quality retained: the router's pass rate against the baseline's, evaluating identical prompts with the same grader on both arms

What counts as a pass:

Where the savings come from, and what they cost​

The split runs opposite to intuition:

• Short, cheap traffic saves the most at no quality cost. GSM8K is identical at a fifth of the price, and HumanEval came out one prompt ahead of the frontier baseline
• Repo-level work saves the least. Those prompts are long enough to dominate absolute spend, and the classifier correctly declines to send them to Haiku
• Quality is given up in two places. MATH-500, where Haiku took 19 of the 40 prompts and the arm finished three behind, and SWE-bench Lite

Auto-routing and prompt caching​

A question we keep hearing: do I have to choose between auto-routing and prompt caching?

No. Session affinity already lets you keep a conversation on the model its first turn picked, so the cache holds for the rest of that conversation. What it costs you is the routing decision on every turn after the first: once a session is pinned, a one-line follow-up stays on whatever model the opening turn earned. Closing that gap is the first item below.

What's next​

• Prompt caching that survives a tier change. Affinity pins a conversation today because switching models means a cold cache. We are testing a background refresher that keeps the prefix warm on every tier, so a session can move without paying the write cost twice
• Routing decisions you can inspect. Surfacing why a request landed on the tier it did, not just which model served it. That same signal feeds back into the classifier, so it improves against real traffic rather than benchmarks

Try it​

model_list:
  - model_name: claude-haiku-4-5           # $1 / $5 per 1M tokens
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-sonnet-5            # $3 / $15
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: claude-opus-5              # $5 / $25
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE:    claude-haiku-4-5
          MEDIUM:    claude-sonnet-5
          COMPLEX:   claude-opus-5
          REASONING: claude-opus-5
        classifier_type: heuristic         # local scoring, sub-millisecond, no API call
      complexity_router_default_model: claude-sonnet-5

Point a client at smart-router and every response carries x-litellm-model-name and x-litellm-response-cost, which is all the instrumentation this study needed. Full reference, including the classifier and tier-boundary knobs, on the Auto Routing docs page.

Security​

38 security fixes shipped this month, including two critical vulnerabilities closed for good. Here is where they landed:

We backport security fixes across the last four minor release lines, so staying current is the whole defense. Upgrade to stay covered.

How we review every PR​

Every pull request runs a security pass before it can merge, and every release runs a full scan before it ships.

• Veria scan. A required check on every PR, built on Veria AI, zizmor, and semgrep. False positives get flagged, never blocking.
• Dependency scan. osv-scanner gates every lockfile change and runs nightly, so a known-CVE dependency can't ship.
• Image scan. Grype checks the runtime image for OS and library CVEs and fails the build on fixable High/Critical findings.

We pay for security reports​

The bug bounty is live.

• Scope: the LiteLLM gateway and SDK.
• Submit through a private vulnerability report on GitHub.
• Triaged by maintainers and the Veria Labs security team.
• github.com/BerriAI/litellm/security

What's next for security​

We're expanding the bug bounty program, hardening the recurring code patterns we keep finding through the stability sprint, and hiring to match. This is a structural investment, not a one-month sprint.

Stability​

We've grown fast, and want to prioritize staying on top of bugs so they don't accumulate. In July we shipped 317 bug fixes.

The target: 95% E2E coverage by end of month​

We're treating end-to-end testing as a first-class citizen. Every new customer journey now ships with its own E2E test, tracked live on the dashboard. The goal is to cover roughly 97 more customer journeys this month, taking us from 309 to 406 of 427 identified journeys, about 95% coverage.

What kinds of fixes shipped​

The headline categories were billing, identity, and MCP.

• Billing accuracy. We closed the gaps where spend used to slip through. Cached and tiered usage on Anthropic and Bedrock is priced correctly now, budgets reset on the right schedule, and partial spend gets captured when a stream drops mid-request.
• Identity and access. Caller identity now resolves once into a single record, so team IDs and spend attribution stay correct, and auth no longer fails open on DB errors.
• MCP reliability. Tools list and call consistently across every auth method now, with per-user credentials and proper OAuth token refresh.
• Resource leaks. Guardrails no longer re-initialize on every request, which kills the runner leaks, latency spikes, and OOMs they used to cause.
• Resilience. Streaming requests recover cost on interruption, the proxy self-heals on dropped DB connections, and OTEL metrics no longer overload Splunk.

Watch our progress live​

Progress is tracked in public on GitHub issue #30484, with the bar set at zero reported regressions by the August 29 release.

• June. 13 bugs triaged and the stability sprint opens. Root-cause analysis begins.
• July. MCP improvements complete, with 80+ fixes shipped, and E2E coverage climbing toward the 95% target.
• August. Fireworks, Cloudflare, and Baseten providers join the E2E suite. Load-testing gates get added to stable releases, with progress reported alongside the August 29 release.

Find our latest stable release here.

Product​

Alongside the security and stability work, we shipped 140 feature commits in July.

• Rust: pyo3 0.29, Python 3.14 support, a Mistral OCR bridge, and /v1/messages on Azure.
• Tokens and budgets: Headroom cuts 60-95% of tokens; budget fallbacks reroute at the cap.
• New models: Claude Sonnet 5 on Day 0, Gemini 3.5 Flash, and Muse Spark 1.1.

Performance​

We've been testing LiteLLM, benchmarked with AIGatewayBench. This is gateway overhead against a local deterministic mock, n=5000 per endpoint on one host.

*Overhead is gateway p99 minus direct-to-mock p99 on the same endpoint. Cost is a footprint estimate from measured CPU and peak RSS at 4 vCPU / 16 GB, and excludes token cost.

Beta v0 is live now for testing. It supports /v1/messages for Azure and all /ocr providers today. See the docs. Next up is /v1/messages for Bedrock Invoke, with support for all known Bedrock auth methods: STS, keys, and IAM.

This is a gradual rollout, one route at a time, each proven in production before the next begins. We're targeting full rollout by December 1st. Same config, same database, same API. There's nothing for you to change.

Get started.

Announcing Autorouter V2​

Autorouter V2 routes each request to the right model by how hard it is. Four complexity tiers, classified by default with rule-based scoring that needs no training data, no API calls, and under 1 ms of latency.

• Simple. "Hello", "What is Python?", "Thanks."
• Medium. "Explain how REST APIs work", "Debug this error."
• Complex. "Design a microservices architecture", "Implement a rate limiter."
• Reasoning. "Think step by step...", "Analyze the pros and cons..."

That default is a heuristic scorer weighing seven signals: token count, code presence, reasoning markers, technical terms, simple indicators, multi-step patterns, and question complexity. The combined score sets the tier (Simple < 0.15, Medium 0.15-0.35, Complex 0.35-0.60, Reasoning > 0.60 or 2+ reasoning markers). V2 adds four ways to make that routing smarter:

1. An LLM classifier. Swap the heuristic for a model call (say anthropic/claude-haiku-4-5, with a configurable timeout) to pick the tier. If the classifier errors, times out, or returns something unparseable, it falls back to the heuristic scorer automatically.
2. Routing that learns. An adaptive bandit mode watches how each conversation actually goes: does the user have to rephrase or correct the model, does it get stuck repeating itself or run out of tool calls, does the user seem satisfied. That live feedback shifts future routing toward the models that are actually working. Quality-vs-cost weighting (we recommend 30/70), the eligible model pool, and a tier-distance penalty are all configurable.
3. Tier escalation. A configured keyword bumps a request one tier higher when it appears in the user's message. Matching is case-sensitive, so a phrase like LITELLM ESCALATE only fires on the exact, shouted form.
4. Semantic keyword matching. Keyword overrides already route requests containing specific terms straight to a tier (for example, "invoice, refund, billing" goes to Medium). Semantic matching extends this with an embedding model and a minimum similarity score, trading a little latency for matches that don't need an exact keyword hit.

We shipped Autorouter V2 to get your feedback on it. Try it today:

• Local CLI
• On Proxy
• Tell us what breaks: GitHub Discussion #32168, or #litellm-autorouter on Discord.

What's next​

Thanks again for all the questions and feedback. We'll keep sharing concrete progress as this work ships, especially as we get closer to the August 29 release.

We're hiring​

LiteLLM is the open-source gateway thousands of teams use to run every model behind one API, from startups to the Fortune 500. We move fast: 140 features and 300+ fixes shipped this month alone.

We're hiring a Security Engineer for the core gateway. Small team, huge surface area, real ownership from day one. Want in, or know someone great? Reach us at recruiting@berri.ai.

Thank you for using LiteLLM. Krrish & Ishaan

Questions? Join the conversation GitHub Discussion #34595

LiteLLM now supports Claude Opus 5 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway. Call it with the same OpenAI-compatible request you already use, and track spend, rate limits, and logging in one place.

What's new in Opus 5​

Key changes (details from Anthropic):

• Much better coding agents: stronger at large refactors, debugging, multi-file features, and finishing end-to-end work without leaving stubs
• Better self-verification: it checks and iterates on its own work more proactively
• Better vision and artifacts: improved screenshots → website, charts and documents, spreadsheets, and slide decks
• Same base API price as Opus 4.8: $5 / MTok input and $25 / MTok output
• Thinking is on by default
• Fast mode: about 2.5x faster, but costs 2x

Before you switch from Opus 4.8​

The swap is not free. Priority Tier and web fetch are both gone on Opus 5, so plan that capacity and that tooling separately. Opus 5's cybersecurity classifiers can also decline a request outright with stop_reason: "refusal" and a category in stop_details, which is worth handling explicitly or routing around with LiteLLM fallbacks.

Sampling parameters (temperature, top_p, top_k), fixed thinking budgets, and assistant message prefill remain unsupported, same as Opus 4.8. Anthropic's Opus 5 migration guide has the full compatibility checklist.

Enabling Opus 5​

Opus 5 ships in the v1.95.0-dev.3 image, cutting later today, but most proxies do not need to upgrade at all. On the default remote cost map, open the Price Data tab under Models + Endpoints in the UI and click Reload Price Data (or POST /reload/model_cost_map as a proxy admin). That refetches pricing and re-registers provider routing in one step, so claude-opus-5 becomes available across Anthropic, Azure, Vertex AI, and Bedrock even on an older version.

The exception is LITELLM_LOCAL_MODEL_COST_MAP=true, which bakes the cost map into the image and puts it out of the Reload button's reach. Pull v1.95.0-dev.3 or later for the bundled Opus 5 metadata:

docker pull ghcr.io/berriai/litellm:v1.95.0-dev.3

Usage​

Pick your provider below. Each tab wires up claude-opus-5 for that provider; the request you send afterward is identical everywhere.

model_list:
  - model_name: claude-opus-5
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.95.0-dev.3 \
  --config /app/config.yaml

model_list:
  - model_name: claude-opus-5
    litellm_params:
      model: azure_ai/claude-opus-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.95.0-dev.3 \
  --config /app/config.yaml

model_list:
  - model_name: claude-opus-5
    litellm_params:
      model: vertex_ai/claude-opus-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.95.0-dev.3 \
  --config /app/config.yaml

model_list:
  - model_name: claude-opus-5
    litellm_params:
      model: bedrock/anthropic.claude-opus-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.95.0-dev.3 \
  --config /app/config.yaml

3. Test it!

The request is the same regardless of which provider you configured above:

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-opus-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Fast Mode​

Opus 5 runs roughly 2.5x faster with speed: "fast", billed at $10 / MTok input and $50 / MTok output (2x the standard rate, down from the 6x premium on Opus 4.6). LiteLLM adds the fast-mode-2026-02-01 beta header and tracks the premium in cost calculations automatically.

curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-opus-5",
  "messages": [
    {
      "role": "user",
      "content": "Refactor this module..."
    }
  ],
  "max_tokens": 4096,
  "speed": "fast"
}'

curl --location 'http://0.0.0.0:4000/v1/messages' \
--header 'x-api-key: sk-12345' \
--header 'content-type: application/json' \
--data '{
    "model": "claude-opus-5",
    "max_tokens": 4096,
    "speed": "fast",
    "messages": [
        {
            "role": "user",
            "content": "Refactor this module..."
        }
    ]
}'

Feedback​

Running Opus 5 through LiteLLM and hitting something unexpected? Share it on GitHub discussion #34517.

We are launching an early beta of the LiteLLM AI Gateway in Rust, and we built AIGatewayBench to measure it against Portkey, Bifrost, and the current LiteLLM Python proxy. Across all four, the LiteLLM Rust gateway has the lowest p99 added latency and the smallest memory footprint by a wide margin: roughly 7x lower overhead and 9x less memory than the next-closest gateway (Bifrost), the lowest cost footprint, and the fastest whole-session times for coding agents. It holds its own on raw sustained throughput and pulls decisively ahead on overhead, memory, and cost.

The results​

Gateway overhead and memory. p99 added latency (log scale) and peak memory. Rust adds about 0.7ms at p99 against 2.3ms (Portkey), 4.5ms (Bifrost), and 257.7ms (Python v1), at 21.8MB peak memory against 90.4MB, 199.1MB, and 329.5MB.

Cost footprint. Estimated USD per one million requests from measured CPU, peak RSS, and sustained throughput on a 4 vCPU / 16 GB instance. Rust lands at about $0.000175, roughly 6x below Bifrost ($0.001008) and Portkey ($0.001042), and far below Python ($0.015354). This is a footprint estimate and excludes token cost, which dominates any real bill.

Throughput efficiency. Sustained requests per second per dollar per hour. Rust reaches about 283,833 against 63,246 (Bifrost) because it holds a similar request rate at a fraction of the CPU and memory. On raw sustained RPS, Rust (~2,814 req/s) edges out Bifrost (~2,744); this chart is efficiency per dollar, not peak throughput.

Agentic coding sessions. Added wall time across a 30-turn Claude Code and Codex-style loop. Rust adds about 0.03s and 0.016s, against 0.13s / 0.047s (Bifrost), 0.12s / 0.09s (Portkey), and 0.97s / 0.24s (Python).

How to reproduce this with AIGatewayBench​

Every gateway points at the same local deterministic Rust mock, so provider latency and network jitter drop out and what remains is the gateway's own cost:

overhead = latency(client -> gateway -> mock) - latency(client -> mock directly)

The gateway's overhead is path A minus path B. Everything except the gateway under test, the mock, the load driver, and the host, is held identical, so the difference is the gateway's own cost. The full harness, per-gateway setup, and raw run data are in AIGatewayBench. Start the mock, start a gateway pointed at it, run a scenario, and regenerate the charts:

python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
cargo run --release -p mock-upstream                 # deterministic upstream
# start a gateway from gateways/<name>/README.md, pointed at the mock
python analyze/make_chart.py                          # charts from results/

A few things to keep honest when reading these numbers:

• Because the upstream is a local mock, the absolute latencies are the gateway's isolated slice, not real-world request latencies. Read them as comparisons under identical conditions.
• Versions: LiteLLM Rust beta, LiteLLM Python v1 (litellm[proxy]), Bifrost v1.6.4, current Portkey OSS. Same mock and load driver on one host.
• Every gateway forwards the Anthropic Messages body with no logging callbacks, spend tracking, or persistence enabled. This isolates forwarding overhead; it is not a full-feature comparison, and enabling those would add cost to every gateway, including ours.
• Single-host, per-scenario runs (for example n=5000 on the overhead panel) without repeated-trial error bars, so treat them as order-of-magnitude differences.
• It is a vendor-run benchmark, so the guardrail is reproducibility: every plotted value is a committed CSV in results/, and the mock and driver are identical across all four gateways.

Conclusions​

For a single chat turn, gateway overhead is noise next to model latency and none of this should change your decision. Where it matters is high request rate against fast responses (embeddings, classification, guardrails) and agentic loops that issue many turns per task, and in the memory and CPU footprint that decides how many pods you run and how close each sits to an out-of-memory kill.

On those axes the LiteLLM Rust gateway is the strongest of the four we tested: the lowest overhead, the smallest footprint, and the lowest cost per unit of throughput, and a raw sustained RPS that edges out the next-best gateway. It is an early beta, streaming and the full feature surface are still landing, and the fastest way to check any claim here is to run AIGatewayBench against your own build. If you want to run the LiteLLM Rust gateway in your stack, sign up for the early beta, and for the architecture behind the migration see Migrating LiteLLM to Rust.

Router plugins are now available on LiteLLM. Each plugin receives the routing context, enriches it, and hands it to the next before the router makes the final decision.

The push came from the autorouter discussion (#32168): teams wanted to layer their own signals (language detection, domain classification, tenant policy, budget caps) onto routing without waiting for each one to land in core. This plugin extension lets teams make these changes while keeping LiteLLM's routing core stable.

Get Started​

A plugin is any object with an async run method that takes the routing context and returns it. Narrow candidate_models to restrict what the router can pick, and write to signals to pass information downstream.

Write one next to your config.yaml in plugins/cheap_first.py:

from litellm.types.router import RoutingContext


class CheapFirst:
    async def run(self, context: RoutingContext) -> RoutingContext:
        cheaper = [m for m in context.candidate_models if "mini" in m]
        context.candidate_models = cheaper or context.candidate_models
        context.signals["cheap-first"] = {"applied": bool(cheaper)}
        return context


cheap_first_plugin = CheapFirst()

Reference it from router_settings.plugins by its dotted path. The proxy loads it from the local file next to config.yaml, or from an installed pip package using the same syntax:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  plugins:
    - plugins.cheap_first.cheap_first_plugin

Start the proxy, and every routing decision now runs through your plugin:

litellm --config config.yaml

Chaining plugins​

Plugins run as a pipeline in list order, and each one sees the mutations and signals the previous plugin left behind. Add a second plugin in plugins/enterprise_only.py that reads the tenant off context.metadata and, for enterprise callers, restricts the pool to the models it left; it can also read the cheap-first signal published upstream:

from litellm.types.router import RoutingContext


class EnterpriseOnly:
    ENTERPRISE = {"acme-corp", "globex"}

    async def run(self, context: RoutingContext) -> RoutingContext:
        tenant = context.metadata.get("tenant", "default")
        if tenant in self.ENTERPRISE:
            context.candidate_models = [
                m for m in context.candidate_models if "gpt-4o" in m
            ]
        # signals from earlier plugins are available here
        cheap = context.signals.get("cheap-first", {}).get("applied")
        context.signals["enterprise-only"] = {"tenant": tenant, "after_cheap": cheap}
        return context


enterprise_only_plugin = EnterpriseOnly()

List both under router_settings.plugins. cheap_first runs first and narrows the pool, then enterprise_only receives that narrowed pool and applies its own policy on top:

router_settings:
  plugins:
    - plugins.cheap_first.cheap_first_plugin
    - plugins.enterprise_only.enterprise_only_plugin

Each plugin only narrows; if any plugin removes every remaining candidate, the request raises rather than silently falling back to the full pool, so a policy in the chain can't be bypassed by an earlier one.

Plugins inside the autorouter​

router_settings.plugins runs the pipeline globally, on every routing decision. You can also scope plugins to the complexity autorouter, where they run inside the tier pick against that tier's actual candidate pool. Put them under complexity_router_config.plugins on the auto-router model:

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: ["gpt-4o-mini"]
          COMPLEX: ["gpt-4o", "gpt-4o-mini"]
        default_model: gpt-4o-mini
        plugins:
          - plugins.cheap_first.cheap_first_plugin

  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

Here the autorouter first classifies the request into a tier, then the plugin filters that tier's pool before a deployment is picked. So a COMPLEX request that classifies into ["gpt-4o", "gpt-4o-mini"] still passes through cheap_first, which keeps only gpt-4o-mini. As with the global pipeline, default_model is not an escape hatch: if a plugin drops every candidate in the tier, the request raises rather than falling back to it.

Two things to know when combining plugins with the complexity router: session_affinity is disabled when plugins are configured, so a mid-session policy change still applies on later turns instead of being skipped by a cached model pin, and adaptive: true alongside plugins raises at config validation, since the bandit selector doesn't consume plugin-narrowed pools yet.

For the full contract, the request lifecycle, and more on scoping plugins to the autorouter's tiers, see the routing plugins docs.

Register your plugin​

We are also making plugins discoverable with a router_plugins.json at the root of the LiteLLM repo.

Here's a sample entry:

  {
    "name": "language-detector",
    "description": "Detects the user's language and publishes a routing signal.",
    "author": "Jean Nuñez",
    "repo": "https://github.com/jeann2013/language-detector",
    "commit": "9e712819269173fc25a16f59ca3e9890f7864ac1",
    "version": "1.0.0",
    "pypi": null,
    "litellm_version": ">=1.94.0",
    "entrypoint": "litellm_plugin_language_detector.plugin.language_detector_plugin",
    "license": "MIT",
    "tags": ["language", "classification", "routing"]
  }

The first community entry is language-detector by Jean Nuñez, which detects the user's language and publishes a routing signal. It pins to a reviewed commit, targets litellm>=1.94.0, and its entrypoint is litellm_plugin_language_detector.plugin.language_detector_plugin; drop that string under router_settings.plugins and it runs. If you've written a plugin, add it to the catalog so others can find it.

Feedback​

If you have any feedback, we'd love to hear it!

Please share your thoughts on the autorouter discussion on GitHub (#32168) or just reach out to me (krrish@berri.ai) with your thoughts on how this can be improved!

Auto Router v2 collapses complexity, semantic, and adaptive routing into a single auto_router/complexity_router. One config now covers heuristic scoring, LLM classification, lexical or semantic keyword rules, and Thompson-sampled tier pools.

The push came from the community. On discussion #32168, users pointed out that all three routing strategies should converge into a single Auto Router. One router with configurable signals and weights keeps the API simple while letting the routing engine evolve internally, instead of forcing you to pick a mode up front.

The operational half came from discussion #32172: predictable beats clever for debuggability. A fixed, versioned mapping from capability class to model is what makes "why did this response cost 4x today" answerable after the fact.

What v2 adds​

One config, all the knobs​

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      drop_params: true
      complexity_router_config:
        tiers:
          SIMPLE:    ["gpt-4o-mini", "claude-haiku-4-5"]   # random-pick pool
          MEDIUM:    gpt-4o                                 # single pin
          COMPLEX:   claude-sonnet-5
          REASONING: gpt-5.5

        # optional: LLM classifier instead of heuristic scorer
        classifier_type: llm
        classifier_llm_config:
          model: claude-haiku-4-5-20251001
          timeout_ms: 2000

        # optional: keyword rules, escalate to highest matched tier
        keyword_tier_rules:
          - keywords: ["hi", "hello", "thanks"]
            tier: SIMPLE
          - keywords: ["kubernetes", "k8s", "istio"]
            tier: REASONING
        semantic_keyword_matching: true
        embedding_model: voyage-3-5
        match_threshold: 0.5

        # optional: append to the built-in technical keyword list
        custom_technical_keywords: [kafka, redis, postgresql, udp, dns]

        # optional: Thompson-sample within the tier's pool
        adaptive: true

        # optional: pin a session to its first-turn model (preserves prompt cache)
        session_affinity: true
        session_affinity_ttl_seconds: 3600

      complexity_router_default_model: claude-sonnet-5

Notes on the new pieces​

LLM classifier goes through the same Router instance, so credentials, budgets, and fallbacks apply. Timeout, empty content, or schema mismatch falls back to the heuristic scorer.

Keyword rules run before the scorer. Multiple matches escalate to the highest tier (SIMPLE < MEDIUM < COMPLEX < REASONING), so rule order does not silently change behavior. Semantic matching uses MAX aggregation (was MEAN), so one strong keyword match is not diluted by other utterances on the tier.

Adaptive turns tier pools into learning pools. Cold requests sample only inside the classified tier instead of collapsing on the cheapest model. Feedback attributes back to the model that actually served the previous turn, even when stateless routing picks a different one this turn.

Session affinity (opt-in) pins the first-turn model for a session and skips reclassification on later turns, so provider-side prompt caches keyed to that model do not get invalidated when a follow-up ("thanks!") would otherwise classify into a different tier (#33126). TTL defaults to 3600s. session_id comes from request metadata.

Decision log emits one greppable line per request:

ComplexityRouter: routing decision cause=complexity_scorer,      tier=SIMPLE,     score=-0.150, signals=['short (7 tokens)', 'simple (what is)'], routed_model=gpt-4o-mini
ComplexityRouter: routing decision cause=literal_keyword_match,  tier=REASONING,                                                                    routed_model=gpt-5.5
ComplexityRouter: routing decision cause=semantic_keyword_match, tier=REASONING,                                                                    routed_model=gpt-5.5
ComplexityRouter: routing decision cause=session_affinity_pin,                                                                                      routed_model=gpt-5.5

Fixes worth calling out​

drop_params, cache_control_injection_points, and any other litellm_params set on the auto router alias itself used to vanish when the router picked a tier. They now merge into the outbound request, without overriding anything the caller passed explicitly (#32974). Same PR fixes an Anthropic /v1/messages to Responses API tool_choice shape bug that broke Bedrock-backed complexity routers (reported in discussion #32168 by @icsy7867).

UI got a working Test Connection per tier (#32950) and required-tier inline validation (#32978).

Try it​

Existing complexity router configs keep working. To try v2, add keyword_tier_rules, classifier_type: llm, adaptive: true, session_affinity: true, or a list value on a tier to your existing complexity_router_config. Full reference on the Auto Routing docs page.

What's next​

Router plugins. From discussion #32168: a pipeline where each plugin receives the routing context, enriches it, and passes it on before Auto Router makes the final call. Plugins do not replace the router; they contribute structured signals (classification, policies, candidate filters, scores) that Auto Router combines.

Concrete end-to-end:

1. User sends a request.
2. Language plugin detects en.
3. Domain classifier labels it coding with 0.93 confidence.
4. Tenant policy limits allowed providers to OpenAI and Anthropic.
5. Budget plugin removes models exceeding the tenant's cost cap.
6. Auto Router picks the best remaining model from the enriched context.

Config sketch:

router_settings:
  plugins:
    - name: language-detector
    - name: domain-classifier
      params:
        provider: openai/gpt-5-mini
    - name: budget-policy
      params:
        daily_limit: 100
    - name: tenant-policy
    - name: custom-python
      path: ./plugins/my_router.py

The initial work landed in #32972; support for plugins on the proxy, and custom plugin files will be next.

Also on the list:

• Escalation ceilings on fallback chains. Per-request cap on escalations plus a cooldown once a key walks the chain N times, so a bad upstream cannot cascade into a bill.
• Attributable decisions. Stamp the routed model and routing-table version on every response, and export structured decision traces (candidates, scores, fallbacks, latency) through the standard logging integrations.

Running Auto Router in production and hitting these? Drop a note on discussion #32168.

Note: If you run Claude Code against Amazon Bedrock through LiteLLM on either v1.91.0 or v1.91.1, upgrade to v1.91.2 or higher.

Summary​

Between July 4 and July 10, proxies running v1.91.0 or v1.91.1 silently broke Anthropic prompt caching for Claude Code sessions routed through Amazon Bedrock's Invoke API. For the customers who reported it, warm-session cache hit rates dropped from roughly 90% to 25-45% and team daily spend rose 2-3x for the same usage. Requests kept returning 200s with correct completions; the only symptoms were the cache miss rate and the bill.

The cause: PR #31364 moved every role: "system" entry in messages into the top-level system field on the Invoke path, which invalidates every cache breakpoint past the tool definitions and system prompt. The fix shipped July 10 in v1.91.2 (#32578, #32831, #32882), with regression tests that fail on pre-fix code.

We own this outcome entirely. The trigger was a poorly documented change in how new Claude models and Claude Code use system messages, but customers run a gateway precisely so they do not have to track provider quirks. Translating requests faithfully, including their caching semantics, is our core job and here we fell short. This post explains exactly what happened, why our testing and review failed to catch it, and what we have changed so this class of regression does not ship again.

Background​

Three facts set up the incident:

1. Claude prompt caching is prefix based:
   1. the pricing of tokens in increasing cost is:
      1. cache read (0.1x)
      2. normal write (1x)
      3. cache write (1.25x for 5m ttl, 2x for 1h ttl)
   2. when Claude Code makes a new request, the Bedrock provider checks if any previous request was a truncated prefix of the current request. If so, it reads from the cache only up to that point.
2. Mid-conversation system messages are new:
   1. On May 28, 2026, Claude Opus 4.8 shipped as the first model accepting role: "system" entries inside messages (docs). This was documented on the Claude API docs on the same day.
   2. Claude Code (v2.1.154) began emitting them on May 28, 2026, with no mention in its changelog.
   3. Bedrock documented the same support by June 9, 2026 at the latest (the first archive.org capture; the page may have appeared as early as May 28).
3. Bedrock has two Anthropic APIs with different rules:
   1. Converse requires all system content in a top-level field; LiteLLM has hoisted it there since December 2024 (#7037).
   2. Invoke takes the native Anthropic Messages format, where models older than Opus 4.8 reject mid-conversation system entries with a 400 and newer models accept them.

What went wrong​

1. After May 28, Claude Code sessions on Bedrock Invoke began failing with 400s mid-session when two things were true:
   1. the model was older than Opus 4.8
   2. the model is served under an alias
      1. Claude Code detects capabilities by looking for version substrings like opus-4-7 or sonnet-4-6 in the model name.
      2. For example, an alias like bedrock-claude contains none, so Claude Code assumes the newest feature set and always emits mid-conversation system messages.
2. An enterprise customer worked around the 400s with a local patch that hoisted every system entry from messages into top-level system, mirroring the Converse behavior, and asked us to upstream it. We shipped it as #31364 in v1.91.0 on July 4.
3. The hoist fixed the 400s but by always hoisting the system entries in messages, that would invalidate the entire messages cache whenever a new mid-conversation system message was written by Claude Code, which we measured to happen every ~3 turns on average.
4. This greatly decreased the cache hit rate and increased the cache write rate, increasing spend.

Detection and response​

On July 8, an affected customer gave a detailed bug report of this regression. We reproduced it ourselves the same day end-to-end.

Three PRs fixed it, all released July 10 in v1.91.2 after extensive end-to-end testing, with regression tests that fail on pre-fix code:

1. #32578 disables mid-conversation system message hoisting on Invoke.
2. #32831 re-enables hoisting on models below Opus 4.8.
3. #32882 disables hoisting on Sonnet 5 and Fable 5, too.

Why our process did not catch this​

1. Testing the original patch was not end-to-end. We validated it with single-turn curl requests showing a 400 become a 200 that we assumed was the shape that Claude Code would use. That was not the case. We did not trace the root cause (we did not yet know mid-conversation system messages existed) or its caching implications, and treated it as a harmless edge case fix.
2. Review lacked the context to object. The human reviewer saw a small compatibility patch with passing tests and no explanation of why Claude Code started sending these kinds of mid-conversation system messages, and our AI review bots did not flag the caching implication either. Nobody in the loop had the info to connect the hoist to cache invalidation.
3. Cost regressions are silent. Every response was a 200 with a correct completion. The only signal was cache-read token counts, which nothing in our CI or monitoring measured.
4. The documentation was incomplete. The feature never appeared in the Claude Code changelog, and as of July 13 the Claude API docs still describe it as Opus 4.8 only (without mentioning Sonnet or Fable 5) and unavailable on Bedrock, which contradicts our empirical measurements.

What we are changing​

• Our e2e suite will gain a scripted multi-turn Claude Code session growing to roughly 250k tokens of context against real Bedrock, asserting cache reads grow monotonically and never collapse (started in #32963).
  • Update (July 21): #32963 merged July 14 with live-Bedrock tests that fail whenever a mid-conversation system message costs a session its cached prefix; #33807 extends the same coverage to Vertex AI and Azure. The full 250k-token session is still in progress.
• We will set up a weekly automated load test that will flag anomalies in spend, cache reads and writes, turn latency, error rates, etc., so cost and performance regressions are detected and fixed before a release.
  • Update (July 21): done in #34166. CI now runs concurrent Claude Code-shaped sessions weekly against real Anthropic and Bedrock Invoke deployments and fails if error rate, warm-turn cache reads, cache write volume, p95 turn latency, or recorded spend leaves its baseline.
• Daily automated diffs of Anthropic's SDKs and docs alert us to new features that need translation support before customer traffic finds them.
• We dogfood LiteLLM internally and will set up monitoring for new request shapes, such as unknown anthropic-beta headers, and the same anomoly detction, which will alert us ahead of a release.
• Bug fixes now have a higher merge bar: validated means reproduced against the real client's traffic on their exact end-user application end-to-end and a complete understanding of the root cause; synthetic requests are not enough.

Known limitations​

1. Converse rejects system entries inside messages at any position, so on bedrock_converse we must still hoist, and Claude Code sessions routed through Converse still lose cached prefix on every mid-conversation system message. If you run Claude Code against Bedrock, route it through the Invoke path (bedrock/invoke/<model>). We are raising the API constraint with AWS.
2. We are testing whether the Vertex AI and Azure paths need equivalent hoisting and will update this post when we have more info.
   • Update (July 21): they do, and #33807 (merged July 20, in the next stable release) closes the gap. We verified live that both providers enforce the same per-model contract as Bedrock Invoke, so both paths now reuse the model-aware hoisting logic: older models return completions instead of 400s and newer models keep the cached prefix byte-identical. Live e2e tests cover both providers.

To every team whose bill went up because of this: we are sorry. The value of a gateway is that this class of provider change gets absorbed by us instead of reaching you, and the tests, monitoring, and process improvements above are how we intend to keep it that way.

1. The MCP Gateway did not have a single class for credential resolution.
2. Pass-through APIs had high memory consumption.

Across the same window we shipped 134 bug fixes in total. This post covers the two big changes first, then the rest of the AI Eng and reliability work, the full breakdown, and what we are doing next.

MCP Gateway: a single credential resolver​

The MCP Gateway connects a user's AI app (Claude Desktop, Cursor, an agent) to the upstream MCP servers it wants to use, and it has to attach the right credential for each upstream.

Before, the MCP Gateway would infer the authentication method a user was trying to use based on the credentials they set. We have now built a credential resolver class that lets a user explicitly specify which auth method they are using. We believe this will significantly drive down a class of bugs users were reporting on MCPs.

Before: infer the auth method from whatever credentials are set​

Why this was bad: there was no single place that decided which credential to attach, and no error when the decision was ambiguous. Inferring from set fields meant two code paths could read the same server and disagree. The precedence order meant adding a field could silently change which credential won. And the silent fallback meant an unhandled case still sent something upstream instead of refusing. Ambiguity resolved to "attach a credential anyway" instead of "stop."

The types of bugs we saw from this:

• Tokens sent to the wrong upstream server.
• Duplicate or stale Authorization headers slipping through.
• MCP requests skipping the normal team, route, and key checks.
• Cached OAuth tokens going stale or crossing between users.
• Upstream URLs and secrets showing up in logs.

After: the user declares the auth method, and it fails closed​

Each mode has its own fully typed config, so there is no guessing from which fields are set and no precedence order. The match is exhaustive, so adding a mode without handling it fails the type checker, and an unhandled case raises instead of quietly attaching no auth.

AI Eng: LLM providers (27 fixes)​

AI Eng was focused on driving down reported bugs. Most of them were on new models (Claude 4.8, Opus 4.8, Bedrock Invoke). Here are the types of bugs we fixed:

Most of these were on Bedrock and the routing layer.

Performance: pass-through memory​

Pass-through APIs had high memory consumption. Large non-JSON pass-through downloads (batch-result files, binary and octet-stream downloads) were buffered whole in memory before being sent on. We changed this to stream the response chunk by chunk, so memory stays flat regardless of file size (#32386). This covers the provider pass-through routes (/vertex_ai/*, /bedrock/*, /openai/*, /anthropic/*, and others) and custom pass-through endpoints.

JSON responses still buffer by design, so spend logging and guardrails can inspect the body.

Two more fixes in the same spirit, don't pay for work nobody needs:

• Prometheus skips budget-metric DB lookups entirely when the gauges are no-ops (nothing is scraping them).
• The complexity router builds its semantic route index once under concurrent cold-start, instead of rebuilding it per request.

By the numbers​

All 134 fixes, by area:

These 134 are every merged fix: PR in the two weeks. One reported ticket often turns into several fix PRs, so this count is higher than the number of tickets in Linear.

Next goal: 95% end-to-end test coverage​

Most of these 134 bugs were caught late, in staging or from a user report. We want to catch them before they merge. We believe by investing in improving our e2e testing coverage we can significantly reduce the number of reported regressions from users on an upgrade.

We are learning from Meta's approach to fixing bugs fast and raising our testing bar.

Appendix​

PRs from this window.

MCP Gateway

Credential resolver:

• #32815 credential class merge (the single typed resolver)
• #32652 stale token invalidation
• #32715 semantic filter fail-closed

Pass-through and delegate credential modes:

• #31989 passthrough / delegate modes
• #32414 passthrough UI enum
• #32556 passthrough call relay
• #32752 configured client for passthrough
• #32735 no DCR persist for passthrough
• #32507 token exchange secret pairing

DCR bridge (LIT-4337):

• #32745 plumbing
• #32747 authorize relay
• #32753 facade
• #32804 UI
• #32527 DCR redirect_uri

Sealed envelope and delegate admission (LIT-4338):

• #32748 envelope module
• #32794 envelope edge consumer
• #32824 delegate admission

AI Eng (LLM providers)

Bedrock:

• #32882 flag Claude 4.8+ entries with supports_mid_conversation_system
• #32831 gate in-place system role messages for Claude Invoke
• #32578 keep mid-conversation system messages for Claude Invoke
• #32658 retain clear_tool_uses context-management edits and emit the beta header
• #32551 honor cache_control ttl on message-level cachePoint blocks
• #32538 preserve cache_control ttl on message-level cache points
• #32840 add jp.anthropic.claude-opus-4-8 to the model cost map
• #32389 resolve regional inference profiles to regional pricing

Anthropic:

• #32874 thread the real provider through capability probes (was pinned to anthropic)
• #32867 translate adaptive thinking/effort for pre-4.6 models
• #32833 strip @version suffix in model lookup

Routing and fallbacks:

• #32859 complexity router keyword tiers (max aggregation, blank-keyword hardening)
• #32943 complexity router logging and auth propagation, index built once
• #32873 fallback rules routing split (bare-Claude coverage, cost-map precedence, legacy schema)

Responses API:

• #32835 raise APIError on in-stream error events
• #32837 preserve reasoning_tokens through chat to responses
• #32034 idempotent response-id encoding (prevents MCP gateway double-encoding)

Cost, Vertex, Rerank:

• #32387 add gpt-realtime-2.1 models with regional uplift
• coerce string tiered-pricing costs and share the tier helper
• #32550 forward realtime health check params (Vertex)
• #32533 log rerank params at debug to stop leaking request content

Performance

• #32386 stream non-SSE pass-through responses instead of buffering in memory
• #32404 stop request params from clobbering merged target query params
• #32834 Prometheus skips budget-metric DB lookups when gauges are no-ops

Agenda​

• Product updates and roadmap progress
• Reliability and security updates
• Open Q&A with the team

Stability​

Our team is focused on stability: improving reliability and reducing regressions across releases. Follow the roadmap here: Stability roadmap.

How to contribute​

Add the topics and questions you'd like us to cover when you register below. We use your responses to set the agenda.

Register​

Register here: LiteLLM July Townhall Form

We will hold the townhall from 7:30 AM to 8:30 AM PT on Zoom.

For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.

LiteLLM now supports the GPT-5.6 family. Route traffic to OpenAI's newest frontier models through the LiteLLM AI Gateway with no code changes.

GPT-5.6 introduces a new naming system where the number identifies the generation and the tier name identifies a durable capability level. gpt-5.6-sol is the flagship for complex reasoning and agentic workloads, gpt-5.6-terra is a balanced model for everyday work with performance competitive with GPT-5.5 at roughly half the cost, and gpt-5.6-luna is the fastest and most affordable tier. Per OpenAI, the family sets a new state of the art on agentic coding (Terminal-Bench 2.1) with broad gains in long-horizon biology and cybersecurity workflows. GPT-5.6 also adds a new max reasoning effort for the deepest single-agent thinking and an ultra mode that coordinates subagents on the most complex tasks.

Usage​

model_list:
  - model_name: gpt-5.6-sol
    litellm_params:
      model: openai/gpt-5.6-sol
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-terra
    litellm_params:
      model: openai/gpt-5.6-terra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5.6-luna
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.93.0-dev.2 \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.6-sol",
    "messages": [
      {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ]
  }'

from litellm import completion

response = completion(
    model="openai/gpt-5.6-sol",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
)

print(response.choices[0].message.content)

# gpt-5.6-terra for balanced, cost-efficient everyday work
response = completion(
    model="openai/gpt-5.6-terra",
    messages=[
        {"role": "user", "content": "Summarize the key ideas in this design doc."}
    ],
)

print(response.choices[0].message.content)

# gpt-5.6-luna for the fastest, lowest-cost tier
response = completion(
    model="openai/gpt-5.6-luna",
    messages=[
        {"role": "user", "content": "Classify this ticket as bug, feature, or question."}
    ],
)

print(response.choices[0].message.content)

model_list:
  - model_name: gpt-5.6-sol
    litellm_params:
      model: azure/gpt-5.6-sol
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: os.environ/AZURE_API_VERSION
  - model_name: gpt-5.6-terra
    litellm_params:
      model: azure/gpt-5.6-terra
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: os.environ/AZURE_API_VERSION

from litellm import completion

response = completion(
    model="azure/gpt-5.6-sol",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
)

print(response.choices[0].message.content)

Responses API​

For agentic and multi-turn workflows, use /v1/responses to preserve reasoning state and output item metadata across turns.

curl -X POST "http://0.0.0.0:4000/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-5.6-sol",
    "input": "Plan and write a Python script that scrapes a webpage and summarizes it."
  }'

Pricing​

Prices are per 1M tokens (USD), shown as short context (≤272K tokens) / long context (>272K tokens).

Global Azure OpenAI deployments (azure/gpt-5.6-*) match these OpenAI list prices. Regional deployments (azure/us/gpt-5.6-* and azure/eu/gpt-5.6-*) carry the standard 10% uplift on the base rate; LiteLLM tracks the difference automatically once you route through the regional model name.

Notes​

• For cost tracking on the GPT-5.6 models, hit the Reload Model Cost Map button in the Admin UI (or POST /reload/model_cost_map). Works on any LiteLLM version v1.76.0 or newer, with no container restart or image upgrade required.
• GPT-5.6 supports reasoning, function calling, parallel tool calls, vision (image input), prompt caching, web search, and structured output; see the OpenAI provider docs for advanced usage.
• The GPT-5.6 family launched in limited preview and OpenAI is expanding availability through the API and Codex; check your OpenAI account for model access.

Claude Code is one of the heaviest consumers of input tokens in a modern engineering org. Long tool loops, large file reads, and MCP catalogs with hundreds of tools push every request toward the top of the context window, and the bill scales with it.

If Claude Code already points at a LiteLLM proxy (via ANTHROPIC_BASE_URL), there are five levers the platform admin can pull to bring that cost down. None of them require a client-side change.

1. Budget windows + budget fallbacks​

Two knobs on the virtual key.

Budget windows cap how much the key can spend inside a rolling time period. Set max_budget (dollars) and budget_duration ("24h", "7d", "30d", etc). LiteLLM resets the counter automatically at the end of every window. You can stack windows too, e.g. $10/day AND $100/month, so one bad afternoon can't burn the whole month:

curl 'http://0.0.0.0:4000/key/generate' \
  --header 'Authorization: Bearer <your-master-key>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "budget_limits": [
      {"budget_duration": "24h", "max_budget": 10},
      {"budget_duration": "30d", "max_budget": 100}
    ]
  }'

Budget fallbacks decide what happens once a per-model budget is exhausted. Instead of erroring at the developer's terminal, attach model_max_budget per model and a budget_fallbacks chain naming the cheaper models to reroute to. The request silently falls to the first fallback still under its own budget:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model_max_budget": {
      "claude-opus-4-8":   {"budget_limit": 20.0, "time_period": "1d"},
      "claude-sonnet-5":   {"budget_limit": 10.0, "time_period": "1d"},
      "claude-haiku-4-5":  {"budget_limit": 5.0,  "time_period": "1d"}
    },
    "budget_fallbacks": {
      "claude-opus-4-8":  ["claude-sonnet-5", "claude-haiku-4-5"],
      "claude-sonnet-5":  ["claude-haiku-4-5"]
    }
  }'

Once the developer burns $20 of Opus in a day, subsequent Opus requests silently reroute to Sonnet; if Sonnet is also tapped out, Haiku picks up. Fallback models without a model_max_budget entry are treated as unlimited.

→ Learn more: Budget Windows · Budget Fallbacks

2. Automatic Prompt Caching​

Claude's prompt cache reads a cache hit for roughly 10% of the price of a fresh input token, but only if the request marks the right message with cache_control. LiteLLM injects that marker for you: point cache_control_injection_points at the system message (or the second-to-last user turn), and every Claude Code call through the proxy carries the checkpoint without any client-side edit.

model_list:
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
      cache_control_injection_points:
        - location: message
          role: system

router_settings:
  optional_pre_call_checks: ["prompt_caching"]

for automatically injecting this in all requests, do this

model_list:
  - model_name: claude-sonnet-4.5-20250929
    litellm_params:
      model: vertex_ai/claude-sonnet-4-5@20250929
      # ...

router_settings:
  default_litellm_params:
    cache_control_injection_points:
      - location: message
        role: system
  optional_pre_call_checks: ["prompt_caching"]

Turning on 'prompt_caching' as a pre call check, means if you run multiple deployments of the same Claude model, LiteLLM will intelligently route to the model deployment which was initially used for the request.

→ Learn more: Auto-Inject Prompt Caching Checkpoints · Claude Code - Prompt Cache Routing

3. Prompt Compression (Headroom)​

Prompt cache trims the static prefix; Headroom trims the dynamic middle. Tool outputs, file reads, database dumps, and RAG payloads get rewritten into a compressed form before they reach the model, and if the model actually needs the original bytes, a retrieve_headroom tool call fetches them on demand. Reported savings run 60-95% on the compressible portion of Claude Code traffic.

Headroom runs as a sidecar container next to LiteLLM. Register it as a pre_call guardrail and either flip default_on: true or attach it to per-developer virtual keys.

guardrails:
  - guardrail_name: headroom-compression
    litellm_params:
      guardrail: headroom
      mode: pre_call
      api_base: https://your-headroom-service
      default_on: true

The developer still exports ANTHROPIC_BASE_URL and runs claude; the only thing they notice is a smaller number on the spend log.

→ Learn more: Headroom guardrail setup guide

4. Defer MCP tools​

A Claude Code session that connects to five or six MCP servers can easily surface a few hundred tools, and every one of those tool schemas ships on every tools/list call. That is pure input-token overhead on a workload where the model uses two or three tools per turn.

Turn on mcp_tool_search_enabled on the virtual key, and LiteLLM replaces the full catalog with two virtual tools, mcp_tool_search and mcp_tool_call. The model searches by keyword, gets the ranked matches back, and calls the one it wants. The token cost of tool listing collapses from hundreds of schemas to two.

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "object_permission": {
      "mcp_tool_search_enabled": true,
      "mcp_servers": ["github", "slack", "linear", "jira"]
    }
  }'

Ranking is token-overlap over name + description, so there is no embedding dependency to run. The access surface does not widen; search only returns tools the key was already allowed to call.

→ Learn more: MCP Tool Search

5. Auto routing​

Send every request to the smallest model that can handle it, so cheap requests never touch the expensive model. LiteLLM ships three flavors: Semantic (embedding match), Complexity (rule-based, zero external call), and Adaptive (learns from live traffic, beta).

Complexity router is the fastest to set up. Point Claude Code at smart-router and it classifies each request into a tier:

model_list:
  # Target models
  - model_name: gpt-4o-mini
    litellm_params:
      model: gpt-4o-mini

  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o

  - model_name: claude-sonnet
    litellm_params:
      model: claude-sonnet-4-20250514

  - model_name: o1-preview
    litellm_params:
      model: o1-preview

  # Complexity router
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-4o-mini
          MEDIUM: gpt-4o
          COMPLEX: claude-sonnet
          REASONING: o1-preview
      complexity_router_default_model: gpt-4o

→ Learn more: Complexity Router · Semantic Auto Routing · Adaptive Router

Stacking the levers​

The five features compose. Budget-based fallbacks bound the total spend regardless of what else you do. Prompt cache checkpoints and Headroom compression each shave a different slice of the request payload before it hits the model. MCP tool search cuts the tool schema overhead at the front of every turn. Auto routing sends every request to the smallest model that can handle it. Turn them on together and the same Claude Code workload runs on a fraction of the input tokens it did before, without touching a single developer machine.

Help us make this better​

We're actively investing in cost optimization across the whole stack. If you've got ideas, on auto routing, better cache heuristics, smarter budget policies, anything, join the discussion at litellm#32172.

LiteLLM now supports Claude Sonnet 5 on Day 0. Use it across Anthropic, Azure, Vertex AI, and Bedrock through the LiteLLM AI Gateway. Call it with the same OpenAI-compatible request you already use, and track spend, rate limits, and logging in one place.

What's new in Sonnet 5​

Sonnet 5 is the most agentic Sonnet model yet, with performance close to Opus 4.8 at a fraction of the price. A few things stand out for teams running it through a gateway:

• Opus-class quality, Sonnet pricing. Anthropic reports Sonnet 5 performs close to Opus 4.8 while costing far less, and is a substantial step up from Sonnet 4.6 on reasoning, tool use, coding, and knowledge work. (details from Anthropic)
• Built to run agents. It plans, drives tools like browsers and terminals, runs autonomously, and checks its own output without being asked, finishing complex tasks where earlier Sonnet models would stop short. Anthropic highlights gains on BrowseComp (agentic search) and OSWorld-Verified (computer use).
• Adaptive thinking only. Sonnet 5 decides how deeply to think on its own. You steer it per request with reasoning_effort or output_config.effort; fixed thinking budgets, temperature, top_p, and assistant message prefill are not supported by the model.
• $3 / MTok input and $15 / MTok output, with prompt caching at $0.30 / MTok (read) and $3.75 / MTok (write). Anthropic is running introductory pricing of $2 / MTok input and $10 / MTok output through August 31, 2026. On Bedrock, the us., eu., au., and jp. inference profiles carry the usual 10% regional premium while global. stays at base price; LiteLLM tracks every variant automatically.
• 1M-token context, up to 128K output tokens.
• One gateway, every surface. Vision, PDF input, computer use, tool calling, prompt caching, adaptive thinking, and structured output, all available across Anthropic, Azure, Vertex AI, and Bedrock with unified spend tracking, logging, and fallbacks.

Enabling Sonnet 5​

Sonnet 5 ships in the v1.92.0-dev.1 image (and every release after it). How you pick it up depends on where your proxy reads pricing from:

• Default (remote cost map): no upgrade needed. In the LiteLLM UI, open the Price Data tab under Models + Endpoints and click Reload Price Data (or, as a proxy admin, POST /reload/model_cost_map). This refetches the latest pricing from LiteLLM's cost map and re-registers provider routing in one step, so claude-sonnet-5 becomes available across Anthropic, Azure, Vertex AI, and Bedrock, even if you're on an older proxy version.

• Running LITELLM_LOCAL_MODEL_COST_MAP=true? The cost map is baked into the image, so the Reload button won't reach it. Pull v1.92.0-dev.1 or later to get the bundled Sonnet 5 metadata:

  docker pull ghcr.io/berriai/litellm:v1.92.0-dev.1
  

Usage​

Pick your provider below. Each tab wires up claude-sonnet-5 for that provider; the request you send afterward is identical everywhere.

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: azure_ai/claude-sonnet-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: vertex_ai/claude-sonnet-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.92.0-dev.1 \
  --config /app/config.yaml

3. Test it!

The request is the same regardless of which provider you configured above:

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Long-context agents burn most of their input budget on repeated tool output, retrieved chunks, and stale scratch state. Headroom intelligently rewrites that content so the model sees the same information at a fraction of the tokens.

If the model needs the full context, LiteLLM will also pass a 'retrieve_headroom' tool to the model, to retrieve the full context from Headroom.

How is it deployed?​

Headroom runs as a sidecar to LiteLLM. Client traffic still hits the LiteLLM gateway; LiteLLM invokes Headroom during the pre_call step, swaps in the compressed messages, and forwards the payload upstream. Clients and the LLM provider never talk to Headroom directly.

The benefit of this is two-fold

• Convenience: Users get 1 api base regardless of if they use prompt compression or not.
• Reliability: If Headroom goes down, your LLM calls are unaffected.

Compression works on both /v1/chat/completions and /v1/messages (Anthropic format), which makes the Claude Code rollout a one-liner for the admin: attach headroom-compression to a virtual key, hand it to the developer, and every request they make through ANTHROPIC_BASE_URL gets compressed automatically. No client-side change, no code diff.

Turn it on per key, per request, or globally via default_on: true. Confirm it ran by checking the x-litellm-applied-guardrails response header or the Guardrails panel in the Logs UI.

Get started: Headroom guardrail setup guide (requires LiteLLM v1.92.x or later; for testing ahead of the stable cut, grab the v1.92.0-dev.1 dev release)

Discussion: GitHub discussion #31816

Thank you to everyone who joined our June town hall.

Three numbers capture the month: 24 security fixes, 94 bug fixes, and 78 feature commits. The sections below break each one down, alongside our public commitment to zero reported regressions and the gradual migration of the LiteLLM gateway to Rust.

Security updates​

Last 4 weeks: by the numbers​

Bug bounty — now live​

We pay for security reports.

• Scope — the LiteLLM gateway and SDK.
• Submit via private vulnerability report on GitHub.
• Triaged by maintainers and the Veria Labs security team.

Automated review on every PR​

Every PR gets a security pass. Look for the Veria scan — it's a required check on every PR, built on Veria AI + zizmor + semgrep. False positives are flagged, never blocking.

What's next for security​

• Invest more in the bug bounty program.
• Improve code patterns during the stability sprint.

Stability updates​

The commitment: zero reported regressions by August 29th​

The goal:

• Close 20 reported bugs in core functionality.
• Fix root causes in 3 high-impact components.
• Ship a public progress report alongside the August 29 release.

94 bug fixes done​

Fixes shipped across five areas:

• Proxy core & resilience — 22 fixes
• UI + Auth / SSO — 22 fixes
• Cost, Budgets & Observability — 21 fixes
• MCP Gateway — 15 fixes
• Streaming / Realtime APIs — 14 fixes

What kinds of fixes shipped:

• Billing accuracy. Closed the gaps where spend slipped through — virtual-key limits are now enforced, and cached and tiered usage on Anthropic and Bedrock is priced correctly.
• Identity & access. Caller identity now resolves once into a single record, so team IDs and spend attribution stay correct and auth no longer fails open on DB errors.
• MCP reliability. Tools now list and call consistently across every auth method, with per-user credentials and proper OAuth token refresh.
• Resource leaks. Guardrails no longer re-initialize on every request, eliminating the runner leaks, latency spikes, and OOMs they caused.
• Resilience. Streaming requests recover cost on interruption, the proxy self-heals on dropped DB connections, and OTEL metrics no longer overload Splunk.

Root causes, not just symptoms:

• MCP authentication — 5 separate code paths, one per auth method, caused inconsistent tool listing and calling. Fix: a single unified code path resolves credentials across all auth methods.
• AI gateway auth — 5+ DB lookups per request to resolve key/user/team identity. Fix: caller identity resolves once into a single record — lookups cut roughly in half.
• UI forms — saving a form could overwrite unrelated fields. Fix: frontend and backend types are 100% in sync from a shared source, so only edited fields change on save.

Public timeline​

Bug triage is open and active on GitHub issue #30484.

• NOW — 20 bugs open in core. Triage active.
• JULY — MCP auth unified to a single code path. AI gateway identity lookups cut in half.
• AUGUST — UI form types synced end-to-end. No more silent field overwrites on save.
• AUG 29 — Public progress report ships with the release. Zero-regression target date.

Product updates​

78 feature commits in June​

Rust

• Rust workspace · Mistral OCR bridge
• OpenAI Realtime translation layer

Sandbox API

• E2B + OpenSandbox
• Unified code execution API

New models/providers

• TinyFish · Fal.ai · Fireworks AI
• Cloudflare Workers AI · MAI-Image-2.5

Performance: moving LiteLLM to Rust​

We're migrating the LiteLLM gateway to Rust, and the early numbers make the case:

Per-request overhead measured at 10 concurrent clients vs. a local mock upstream; throughput and memory under sustained load at 50 concurrent clients. Reproducible harness checked in.

How the migration works: a staged rollout, moving piece by piece from a pure Python SDK + FastAPI proxy, to Python driving Rust transforms via PyO3, to a FastAPI shell with pure Rust on the hot path, to an all-Rust async server (axum).

A gradual rollout — one route at a time, proven in production before the next begins. Same config, database, and API: nothing for you to change.

• Aug 15 — OCR routes: Mistral first, then all OCR.
• Sep 1 — /messages, then /chat/completions.
• Sep 15 — The router: load balancing, fallbacks, retries, cooldowns.
• Dec 1 — The full server: FastAPI thin shell, then pure Rust (axum).

Announcing our version policy​

Going forward, we'll maintain only the four most recent stable minor releases. This takes effect next Monday, June 29th. Our focus is ensuring stability on the most up-to-date product offerings — bookmark our Release Notes to stay current.

What's next​

Thank you again for all the questions and feedback. We'll keep sharing concrete progress updates as these efforts ship — especially as we approach the August 29 zero-regression milestone.

Hiring​

We are actively hiring across several roles — apply here if you're interested!

Thank you for using LiteLLM - Krrish & Ishaan

The OpenAI Responses and Chat Completions APIs let you declare a code_interpreter tool and the model runs Python inside an OpenAI-hosted container. That container is opaque, billed by OpenAI, and the code (often customer data) leaves your perimeter. LiteLLM now let's you intercept that tool call and runs it in a sandbox you control. The client request is unchanged.

Available starting LiteLLM v1.91.0.dev1. Check here for releases.

How the swap works​

Register a sandbox tool, enable the interceptor, then call the model the way you already do. When the model emits a code_interpreter tool call, LiteLLM creates a sandbox (E2B or OpenSandbox), executes the generated code, feeds the result back into the loop, and tears the sandbox down on completion. The response shape stays compatible with OpenAI's native code_interpreter_call.

Two backends are supported today: E2B for a managed sandbox, and OpenSandbox for self-hosted Docker-backed execution when the code or data cannot leave your network.

SDK​

import os, litellm
from litellm.sandbox.sandbox_tools import register_sandbox_tools
from litellm.integrations.code_interpreter_interception.handler import (
    CodeInterpreterInterceptionLogger,
)

os.environ["E2B_API_KEY"] = "e2b_..."
os.environ["OPENAI_API_KEY"] = "sk-..."

register_sandbox_tools([
    {
        "sandbox_tool_name": "my-e2b",
        "litellm_params": {
            "sandbox_provider": "e2b",
            "api_key": "os.environ/E2B_API_KEY",
        },
    }
])

litellm.callbacks = [
    CodeInterpreterInterceptionLogger(sandbox_tool_name="my-e2b")
]

response = await litellm.aresponses(
    model="openai/gpt-5",
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    input="Product of first 6 primes. Just the number.",
)
print(response.output_text)

import os, litellm
from litellm.sandbox.sandbox_tools import register_sandbox_tools
from litellm.integrations.code_interpreter_interception.handler import (
    CodeInterpreterInterceptionLogger,
)

os.environ["E2B_API_KEY"] = "e2b_..."
os.environ["OPENAI_API_KEY"] = "sk-..."

register_sandbox_tools([
    {
        "sandbox_tool_name": "my-e2b",
        "litellm_params": {
            "sandbox_provider": "e2b",
            "api_key": "os.environ/E2B_API_KEY",
        },
    }
])

litellm.callbacks = [
    CodeInterpreterInterceptionLogger(sandbox_tool_name="my-e2b")
]

response = await litellm.acompletion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Product of first 6 primes. Just the number."}],
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    max_agentic_loops=4,
)
print(response.choices[0].message.content)

The native code_interpreter tool is rewritten before it reaches OpenAI; on the chat path it becomes a litellm_code_execution function tool and LiteLLM appends each sandbox result as a role: tool message until the model returns a final answer.

Proxy​

Same swap behind the AI gateway, with no client-side change.

model_list:
  - model_name: gpt-5
    litellm_params:
      model: openai/gpt-5
      api_key: os.environ/OPENAI_API_KEY

sandbox_tools:
  - sandbox_tool_name: my-e2b
    litellm_params:
      sandbox_provider: e2b
      api_key: os.environ/E2B_API_KEY

litellm_settings:
  callbacks: ["code_interpreter_interception"]
  code_interpreter_interception_params:
    sandbox_tool_name: my-e2b

The OpenAI SDK keeps working unchanged. Point it at the proxy, declare code_interpreter, and the gateway handles the rest.

from openai import OpenAI

client = OpenAI(api_key="sk-1234", base_url="http://localhost:4000/v1")

response = client.responses.create(
    model="gpt-5",
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    input="Product of first 6 primes. Just the number.",
)
print(response.output_text)

To run fully on-prem, swap the sandbox_tools entry to OpenSandbox:

sandbox_tools:
  - sandbox_tool_name: my-opensandbox
    litellm_params:
      sandbox_provider: opensandbox
      api_base: os.environ/OPEN_SANDBOX_API_BASE
      api_key: os.environ/OPEN_SANDBOX_API_KEY

OpenSandbox runs sandboxes locally with egress denied by default; flip allow_internet_access=True or pass an explicit network_policy when the code needs the network.

Why route it through your own sandbox​

You keep the OpenAI client contract while owning the execution layer. The generated code and any uploaded data stay inside the sandbox you operate, billing for execution stops going to OpenAI, and the same setup works for Responses and Chat Completions across any model the gateway routes to. Streaming, forced tool_choice, and concurrent requests are isolated per request and cleaned up on completion.

Full reference is in the sandbox docs.

Last Updated: June 2026

Over the past year, we have heard the same thing from our users and our community: they want the fastest, most lightweight AI gateway they can run. We have heard you. We are addressing it by moving LiteLLM to Rust, and committing to sub-1ms overhead with a sub-100MB memory binary you can deploy. By the end of this migration, you will get a pure Rust server that can serve 100% of your AI traffic, with every hot path operation, including auth and rate limiting, running in Rust.

The reason it matters: under real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time. Today the LiteLLM Python proxy peaks around 359MB of memory under load, and that cost multiplies across every pod, region, and retry you run.

We are already seeing the payoff in benchmarks. The Rust gateway serves about 15x the throughput (453 to 6,782 requests per second) on about 11x less memory (359MB to 32MB), and cuts per-request overhead from about 7.5ms on the Python path to about 0.05ms, well under the 1ms we commit to.

What you get​

You deploy a single Rust binary. It uses about 65MB of memory, gateway overhead stays under 1ms, and nothing in your setup changes: same config.yaml, same database, same client API, same providers. You keep LiteLLM's coverage of 100+ LLM providers behind one OpenAI-compatible API, with /chat/completions, /messages, /responses, and every other LLM endpoint LiteLLM supports today, now as the fastest and most lightweight LLM gateway you can self-host.

This is not a v2 and not a rewrite. There is no new major version to migrate to and nothing for you to change. The runtime under the hot path gets faster and lighter while your config stays exactly where it is.

We ship this the careful way. Each route moves to Rust only after it passes our full parity and end-to-end test suite, and it runs in production before the next route starts. Stability is the priority, and we target zero regressions on every release.

How fast is the LiteLLM gateway? A throughput, overhead, and memory benchmark​

Per-request overhead. We built a small harness: a mock upstream, a thin Rust forwarding gateway (axum), the same forwarding path running through LiteLLM today (litellm.acompletion over uvicorn), and a load client that times each request in microseconds. At 10 concurrent clients against the same mock, the Rust gateway adds about 0.05ms of overhead per request; the LiteLLM Python path adds about 7.5ms. That is roughly 150x lower, and well under the 1ms we commit to.

Sustained load. Against the current LiteLLM Python proxy on the same /v1/responses workload at 50 concurrent clients, the Rust path served about 15x the throughput on about 11x less memory.

The overhead harness (mock, gateway, load client) is checked in next to this post under benchmark/, and the summarized numbers are in rust_proxy_benchmark_results.csv, so you can reproduce the sub-1ms result. This measures the gateway forwarding path (request transform, forwarding, response handling), not a full production workload.

What stays the same​

Nothing you depend on changes. The migration is invisible from the outside:

• Your Python SDK keeps the exact same interface; the same calls now run on Rust bindings underneath.
• Your config.yaml is unchanged.
• Your database and schema are unchanged.
• Your client API and request/response shapes are unchanged.
• Your providers, routing, and keys are unchanged.

You get lower memory and lower overhead, and you do nothing to get it.

How the migration works​

If you just want the outcome, you have it above. The rest of this post is for engineers who want to see how we move the gateway to Rust without breaking anything.

The core idea is a clean split. We build one Rust core that only transforms data: it turns your request into a provider request, turns the provider response back, transforms stream chunks, counts tokens, and normalizes errors. It never opens a socket, reads a secret, or writes to your database. The host process does all of that. That separation is what lets us put Rust into production without rewriting the server, because Python keeps doing the I/O while Rust takes over the translation.

One route at a time, proven in production​

We never flip a whole endpoint at once. For each route we prove one provider first, roll it out to every provider on that route, and only then start the next route. The smallest, lowest-risk route goes first.

In Stage 1 the server does not change shape. Python still serves traffic and does the I/O, but hands translation to the Rust core through a flag-gated binding, per provider. A parity check enforces identical output before any provider turns on, and if the flag is off the existing Python path runs unchanged.

The routes move in order of risk:

• OCR first. Start with Mistral OCR, the smallest surface: no streaming, tiny schema, few params. Once it matches the Python output byte for byte in production, roll out to all OCR providers, then move the route into the Rust core. Integration risk is retired here before any larger endpoint moves.
• /v1/messages next. This adds streaming: SSE parsing, chunk emission, usage accounting, token cost. One provider first, then all, then the route into Rust.
• /chat/completions after that. The largest surface, taken on only once streaming is proven: tools, function calling, multimodal, and the full optional-param matrix.
• Major providers. Azure, then Bedrock, then Vertex, by traffic volume. Auth-coupled providers get signed headers from the host (boto3 / google-auth first, native Rust later). Long-tail providers keep running on Python.

Onto a Rust server​

Once the routes run on Rust, the router moves too: routing, fallbacks, retries, and cooldowns, with state in Redis. Then the server itself moves in two steps.

• FastAPI as a thin shell. FastAPI still terminates HTTP and runs auth, rate-limit, and callbacks, but the entire forwarding path is a single call into Rust.
• Pure Rust server. A native server (axum / hyper) runs the forwarding path with no Python on the hot path. Your custom Python plugins (auth, guardrails, callbacks, SSO) keep working in an optional sidecar, so nothing breaks. We roll it out with shadow traffic and a percentage cutover.

The end state is a pure Rust data plane. Customer Python plugins keep running in the sidecar, so it is non-breaking. Removing Python entirely would require porting plugins to a Rust or WASM interface, which is a breaking change we are deferring.

Why this order​

• The OCR route retires integration risk on the smallest surface.
• /v1/messages retires streaming risk before the largest parameter set.
• /chat/completions is taken on only after streaming is proven.
• By the time the server moves, the core, providers, and router are already running in production through the SDK, so the server work is mostly plumbing.

Every step ships to real users before the next begins, with the parity check as the gate.

Timeline​

We move one function at a time, smallest first, and only after each step passes our test suite.

Frequently asked questions​

Is LiteLLM the fastest LLM gateway?​

That is the goal of this work. With the Rust hot path, LiteLLM targets sub-1ms gateway overhead and a sub-100MB binary, matching compiled-language gateways while keeping coverage of 100+ providers behind one OpenAI-compatible API. In our benchmark the Rust gateway adds about 0.05ms of overhead per request, versus about 7.5ms for the LiteLLM Python path today, and serves 6,782 requests per second under load at 31.7MB peak memory. Gateway overhead is usually a small fraction of total model latency, so it matters most for high-throughput, low-latency workloads like classification and embeddings at scale.

Is LiteLLM slow?​

Gateway latency and throughput depend on how you deploy the proxy: worker count, concurrency settings, and whether logging callbacks run on the hot path. Tuned, the Python proxy serves production traffic across hundreds of providers today. Moving the hot path to Rust pushes the floor lower still: in our reproducible benchmark the Rust gateway adds about 0.05ms of overhead per request, versus about 7.5ms for the LiteLLM Python path, and serves 6,782 requests per second at 31.7MB peak memory.

Is LiteLLM limited by the Python GIL?​

The GIL only matters for CPU-bound work on the request path, and the gateway is mostly I/O. LiteLLM scales today by running multiple workers. The Rust migration removes the question for the hot path: request transforms, streaming, and routing run in the Rust core and router, outside the GIL, with no first-party Python on the forwarding path in the end state.

How much memory does the LiteLLM gateway use?​

The Python proxy peaked at 358.9MB under our load test. The Rust end state targets roughly 65MB. Lower, bounded memory is the main reason for this work: it reduces the high-CPU and OOM failures that show up under concurrent load.

Are these benchmarks reproducible?​

Yes. The overhead harness (a mock upstream, a thin Rust gateway, and a load client that times each request in microseconds) is checked in under benchmark/, alongside the summarized CSV. Same upstream and payload for both runtimes; the only variable is Python versus Rust.

Will the Rust gateway be a breaking change?​

No. Config, database schema, and the client API contract stay the same. The runtime under the hot path changes gradually, route by route, behind passing parity and end-to-end tests.

We are hiring Rust engineers​

We are building this with a small team and looking for Rust engineers who want to work on the hot path of an AI gateway that serves 100+ providers. If that sounds like you, come build it with us.

References​

• How Datadog migrated their static analyzer from Java to Rust
• How GitGuardian migrated the heart of their platform to Rust
• LiteLLM AI Gateway, full feature overview
• Load balancing and routing across 100+ LLM providers

Why we're doing this​

Maintaining older lines means carrying every fix back to keep them all in parity. That overhead grows with the number of lines we keep alive, not the number of fixes we make. Our focus is ensuring the most up-to-date product offerings are stable and working for you. Because of this, LiteLLM is focusing on the four most recent stable minor lines going forward.

How the rolling window works​

This shift in focus takes effect Monday, June 29, 2026.

A minor line is a release series written as 1.89.x, covering every patch in it: 1.89.0, 1.89.1, 1.89.2, and any later ones. We support the four most recent lines and every patch inside each of them.

Today the four supported lines are 1.89.x, 1.88.x, 1.87.x, and 1.86.x. Everything 1.85.x and earlier has reached end of life and will no longer actively receive updates. The window rolls forward: when 1.90.x ships, 1.86.x rolls out and the supported set becomes 1.90.x, 1.89.x, 1.88.x, and 1.87.x. With a new line about every week, that works out to roughly a month of coverage per line.

What this means for you​

To stay supported, pin to a line and take its patches, then move up before it ages out. Patching within a line is a drop-in; moving up a line is where you'd check the release notes for changes. Enterprise customers who need longer coverage can reach out, and for rare high-severity issues we'll use our judgment and may patch outside the window.

How to stay current​

The best way to stay up to date on these changes is to bookmark our release notes. We update it as new versions ship, so you can see the latest stable line and the three behind it that are still supported.

Why this matters​

Semantic caching stores responses by the meaning of a prompt rather than an exact string match, so a reworded request can still hit the cache and skip a paid model call. Until now LiteLLM's semantic cache was built on RedisVL, which depends on RediSearch's FT.* vector API. RediSearch is not available on Redis OSS or on ElastiCache for Redis OSS, which left teams standing up Redis Stack or Qdrant just to get semantic caching. With Redis moving to a source-available license, more teams are standing up Valkey instead, and ElastiCache for Valkey is a common managed target.

Valkey ships vector search through the valkey-search module, and ElastiCache for Valkey exposes it. LiteLLM's new backend talks to valkey-search directly over the Redis protocol, so semantic caching on ElastiCache for Valkey works without RediSearch, Redis Stack, or Qdrant in the path.

How it works​

The valkey-semantic backend builds its own vector index from the field types valkey-search supports, a tag field that isolates each cache key's scope and an HNSW vector field for the prompt embedding, then runs a KNN query at lookup time and returns the cached response when the cosine similarity clears your threshold. Prompt extraction, embedding generation, and response handling are shared with the existing Redis semantic cache, so behavior matches the Redis path including per-request scope isolation. Connections resolve from VALKEY_HOST, VALKEY_PORT, and VALKEY_PASSWORD, falling back to the REDIS_* equivalents, and passwordless clusters are supported for IAM or no-auth setups.

Get started​

Add the cache to your config.yaml:

litellm_settings:
  cache: True
  cache_params:
    type: valkey-semantic
    host: os.environ/VALKEY_HOST
    port: os.environ/VALKEY_PORT
    valkey_semantic_cache_embedding_model: openai-embedding
    similarity_threshold: 0.8

For ElastiCache with encryption in transit, pass a rediss:// URL through cache_params.redis_url instead of host and port. To try valkey-search locally, the bundled image has the module ready:

docker run -d -p 6379:6379 valkey/valkey-bundle:8.1

See the caching docs for the full setup, including the SDK usage and the parameter reference.

We are hosting our June townhall on Thursday, 25 June at 7:30 AM PST.

Agenda​

• Product updates and roadmap progress
• Reliability and security updates
• Open Q&A with the team

Stability​

Our team is focused on stability: improving reliability and reducing regressions across releases. Follow the roadmap here: Stability roadmap.

How to contribute​

Add the topics and questions you'd like us to cover when you register below. We use your responses to set the agenda.

Register​

Register here: LiteLLM June Townhall Form

We will hold the townhall from 7:30 AM to 8:30 AM PST on Zoom.

For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.