Agent simulations

Overview

Simulations evaluate your agent by running it against an LLM-driven simulated user that plays out a scenario from start to finish, then judges whether the agent met your expectations. Unlike the test framework, which asserts on individual turns, simulations evaluate the conversation as a whole: the simulated user pursues a goal, your agent responds with its real logic and tools, and the result includes a pass or fail verdict and the full transcript.

Simulations run in two modes. In text mode the simulated user exchanges text with your agent, testing your LLM, tools, and conversation logic. In audio mode it speaks and listens over a real audio track, testing the whole STT-LLM-TTS pipeline and measuring how the call sounds.

Run a few scenarios while you iterate, or a larger batch to catch regressions before you ship. Simulations execute in parallel on LiveKit Cloud, up to your project's maximum concurrency limit.

Requirements

Before you begin, make sure you have the following:

• LiveKit CLI v2.16.4 or later for Python, or v2.16.7 or later for Node.js. Audio simulations require v2.18.3 or later. See Install the CLI to install or upgrade lk.
• LiveKit Agents 1.6.6 or later (Python), or @livekit/agents 1.6.0 or later (Node.js).
• A LiveKit Cloud project. Simulations run on LiveKit Cloud using your project's credentials, so the CLI must be authenticated to a project.

How it works

A simulation run has three components:

• The simulated user. An LLM follows the scenario's instructions (a persona and a goal for the simulated user) and chats with your agent until the conversation reaches a natural end.
• Your agent. By default, the CLI starts your real agent as a local worker and dispatches jobs for the simulated rooms to it. Your entrypoint, tools, and conversation logic all run unchanged. To run against an agent that's already running, use --agent-name.
• The judge. When the conversation ends, the simulator judges the transcript against the scenario's agent_expectations and records a verdict. You can layer your own check on top to grade against real end state. See Grade on final state.

By default, the simulated user interacts over text (see Text mode), so a run exercises your LLM, tools, and logic without the STT and TTS pipeline. This makes runs fast, cheap, and deterministic enough to put in CI.

Run a simulation

Run simulations from your agent's project directory with the LiveKit CLI:

lk agent simulate -n 10

With no scenario file, the CLI generates scenarios from your agent's source. Because this uploads your code to LiveKit Cloud for the generator to read, the CLI asks you to confirm first. It then starts your agent, dispatches the generated scenarios, and reports results live, with a link to the run in the dashboard.

Command options

lk agent simulate accepts the following options:

The entrypoint is auto-detected, or you can pass it as a positional argument: lk agent simulate agent.py for Python, or lk agent simulate agent.ts for Node.js.

Run against a live agent

By default, lk agent simulate starts your agent as a local worker, registers it under a temporary name, dispatches jobs for the simulated rooms to it, and stops the worker when the run finishes.

To run against an agent that's already running instead of spawning one locally, pass --agent-name:

lk agent simulate --scenarios scenarios.yaml --agent-name my-agent

Pass the agent's registered name, or an empty string ("") to target the project's default agent that auto-joins every room. Running against a live agent requires --scenarios, since there's no local source to generate scenarios from.

Iterate with scenarios.yaml

Generating from source is a fast way to bootstrap a set of scenarios, but the real workflow is iterative: capture the scenarios you care about in a scenarios.yaml file, run them, refine the ones that surface bugs, and re-run. A checked-in scenario file is the source of truth. It's reproducible, reviewable, and runnable in CI.

lk agent simulate --scenarios scenarios.yaml

A scenario file is a named group of scenarios:

name: Room booking
scenarios:
  - label: Book a king room for one night
    instructions: >
      You are Jordan Reyes (email jordan.reyes@example.com, phone 5550142).
      Book a king room for the night of 2026-06-09, checking out the 10th,
      just you. No breakfast, no late checkout. Pay with the card ending 4242.
    agent_expectations: Room booked successfully
    tags:
      feature: room_booking
    userdata:
      guest:
        first_name: Jordan
        last_name: Reyes
      room_type: king
      available_rooms:
        "2026-06-09":
          - king
          - queen
      expected_state:
        booking:
          room_type: king
          check_in: "2026-06-09"
          check_out: "2026-06-10"

Each scenario has the following fields:

Connect scenarios to your agent

To use scenario userdata, your agent must read it from the simulation context. In your entrypoint, call ctx.simulation_context() (Python) or ctx.simulationContext() (Node.js) to detect a simulated run and use deterministic, per-scenario state. It returns a SimulationContext during a simulation, or None (Python) / undefined (Node.js) in production. Userdata arrives as decoded JSON with the keys exactly as written in your scenarios.yaml file. For example, read room_type rather than roomType.

The simulation context is available immediately from the job's dispatch attributes, so you can read it as soon as your entrypoint runs. In most agents, that means reading it right after you connect:

from livekit.agents import AgentServer, JobContext, mock_tools

server = AgentServer()


@server.rtc_session(on_simulation_end=on_simulation_end)
async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    tool_mocks = {}
    if sim := ctx.simulation_context():
        # Seed deterministic state from the scenario's userdata.
        inventory = build_fake_inventory(sim.userdata()["available_rooms"])
        tool_mocks = build_tool_mocks(inventory)
    else:
        inventory = production_inventory()

    session = AgentSession(userdata=Userdata(inventory=inventory), ...)

    # Mock the agent's tools under simulation so runs are reproducible. The LLM
    # still sees the real tool schemas; only execution is intercepted.
    mock_tools(MyAgent, tool_mocks, session=session)
    await session.start(agent=MyAgent(), room=ctx.room)

import { type JobContext, defineAgent, inference, tool, voice } from '@livekit/agents';
import { z } from 'zod';

export default defineAgent({
  entry: async (ctx: JobContext) => {
    await ctx.connect();

    // Under a simulation, seed deterministic state from the scenario's userdata;
    // otherwise use your real backend. The tool below reads `inventory`, so the
    // LLM sees the same tool schemas either way. Only the data changes.
    // Userdata values are typed `unknown`, so `buildFakeInventory(rooms: unknown)` validates.
    const sim = ctx.simulationContext();
    const inventory = sim
      ? buildFakeInventory(sim.userdata()['available_rooms'])
      : productionInventory();

    const agent = voice.Agent.create({
      instructions: 'You are a hotel booking assistant.',
      tools: [
        tool({
          name: 'getAvailability',
          description: 'List room types available on a date.',
          parameters: z.object({ date: z.string() }),
          execute: async ({ date }) => inventory.roomsFor(date),
        }),
      ],
    });

    const session = new voice.AgentSession({
      llm: new inference.LLM({ model: 'openai/gpt-4.1-mini' }),
      // Plus your usual stt, tts, and turn detection.
      // A text simulation drops them automatically.
    });

    await session.start({ agent, room: ctx.room });
  },
});

Checking for a simulation context keeps the production code path unchanged. In a real session, the context is absent, so the agent connects to its real backends. The front-desk example  (Python) seeds a deterministic calendar this way, and the hotel receptionist example  (Python) seeds a SQLite database. Both are useful references for wiring simulations into a production-shaped agent.

Grade on the final state

The simulator's verdict is an LLM evaluation of the conversation. That isn't always sufficient, for example, a polished conversation can still book the wrong room. Register an on_simulation_end (Python) or onSimulationEnd (Node.js) callback to validate your agent's final state and fail the simulation if it doesn't match the expected result:

from livekit.agents import SimulationContext


async def on_simulation_end(ctx: SimulationContext) -> None:
    expected = ctx.userdata().get("expected_state")
    if not expected:
        return  # grade on the conversation alone

    session = ctx.job_context.primary_session
    if not booking_matches(session.userdata.db, expected):
        ctx.fail(reason="final DB state diverged from the expected booking")

import { type JobContext, type SimulationContext, defineAgent } from '@livekit/agents';

// Node.js has no equivalent of Python's `ctx.job_context.primary_session`, so
// keep a handle to the state you want to grade, keyed by the job.
const gradedState = new WeakMap<JobContext, Db>();

export default defineAgent({
  entry: async (ctx: JobContext) => {
    const sim = ctx.simulationContext();
    const db = sim ? seedDb(sim.userdata()) : productionDb();
    if (sim) gradedState.set(ctx, db);
    // Start the session as usual; the agent's tools read and write `db`.
  },
  onSimulationEnd: (ctx: SimulationContext) => {
    const expected = ctx.userdata()['expected_state'];
    if (!expected) return; // grade on the conversation alone

    const db = gradedState.get(ctx.jobContext);
    if (db && !bookingMatches(db, expected)) {
      ctx.fail('final DB state diverged from the expected booking');
    }
  },
});

Key points:

• Your check can only fail a simulation. It can't override a failed simulator verdict. The final result is the logical AND of the simulator's verdict and your check. Calling ctx.fail() fails a simulation the simulator passed, but it can't pass one the simulator failed. If you don't call ctx.fail(), the simulator's verdict stands.
• Use ctx.simulator_verdict (Python) or ctx.simulatorVerdict (Node.js) to inspect the simulator's decision, including its success flag and reason. It's only available inside the callback.
• Access your agent's final state to compare it against the expected state defined in the scenario's user data. In Python, read ctx.job_context.primary_session for the room, session, and user data your agent accumulated. Node.js has no equivalent accessor, so keep a reference to the state you want to grade, such as the backend you seeded from userdata(), and read it in the callback as shown above.

This pattern turns simulation into an evaluation by checking both the conversation and the resulting application state. A simulation passes only if both are correct.

Text mode

Simulations run in text mode by default. The simulated user exchanges text with your agent, so the run tests your LLM, tools, and conversation logic without the STT and TTS pipeline. Text mode is faster, cheaper, and more deterministic. Use text mode for iteration and CI.

Under a text simulation, the framework automatically disables STT, TTS, VAD, and audio input and output, so your agent runs its LLM and tools unchanged without any audio setup.

To check which mode a simulation is running in, read sim.simulation_mode (Python) or sim.simulationMode (Node.js) from the SimulationContext. It returns a SimulationMode enum value. Compare it against SimulationMode.SIMULATION_MODE_TEXT (Python) or SimulationMode.TEXT (Node.js). An unspecified mode resolves to text.

Audio mode runs the same scenarios through your agent's full media pipeline and scores the call itself. To learn more, see Audio simulations.

Audio simulations

In audio mode, the simulated user speaks your scenario aloud, listens to your agent, and interrupts as a real caller would. Your agent runs its full STT-LLM-TTS pipeline against that audio, and the run scores the aspects that only speech exposes:

• Turn-taking. Whether the agent starts speaking before the caller finishes, or leaves a caller who has finished waiting.
• Interruption handling. Whether the agent yields to a barge-in and correctly distinguishes a brief acknowledgment from a turn.
• Transcription. Numbers, spelled-out names, addresses, and confirmation codes, in both directions.
• Perceived speed and speech quality. The latency the caller hears, and whether names and amounts are pronounced correctly.

Run a scenario file in audio mode with the audio subcommand:

lk agent simulate audio --scenarios scenarios.yaml

How an audio simulation runs

Scenarios, the judge, and the pass or fail verdict work the same way in both text and audio modes. Audio mode adds the following behavior:

• The simulated user joins the room as a participant. It publishes an audio track and subscribes to your agent's, so your agent sees a caller rather than a text stream.
• Your configured audio models run. STT, TTS, and VAD use the models configured for your agent, and their providers are billed accordingly. An agent with no STT or TTS configured has nothing to exercise in audio mode.
• The run executes in real time at normal inference priority. This means measured response times match production. Text runs are dispatched to LiveKit Inference  as low-priority batch load because nothing is waiting on the results.
• Usage is metered in audio turns. Audio turns are tracked separately from text turns.

What an audio run measures

Alongside the verdict, an audio run measures call quality per turn and aggregated over the run, so you can track real-world quality over time rather than correctness alone. These metrics appear in the dashboard and in the exported JSON.

Responsiveness. The primary metric is the end-to-end latency the caller heard, reported at p50, p95, and p99. A positive value indicates a gap before the agent responded, while a negative value means the agent talked over the caller. Track this separately from the agent's own reported latency. The agent measures when it started producing audio, while the caller measures when they heard it. The difference between the two is what the user perceives. The run also breaks the pipeline down stage by stage: STT and endpointing delay, LLM time-to-first-token and time-to-first-sentence, tokens per second, and TTS time-to-first-byte. This helps identify the stage responsible for slow responses.

Turn-taking. A turn-taking score with the specific failures behind it: end-of-turn mispredictions where the agent started speaking before the caller finished, time to yield after a barge-in, false interruptions where the agent stopped when there was no interruption, the share of overlapping speech, awkward silences after a natural pause, and caller turns that the agent never answered.

Speech accuracy. Word and character error rates in both directions: what the agent heard compared with what the caller said, and what the caller heard compared with what the agent said. Key entities, such as names, IDs, confirmation codes, and amounts, are scored separately, with recall distinguishing between an entity the agent never recognized and one it recognized but later lost.

Conversation quality. Accuracy and experience scores, conciseness, and whole-call issue flags such as unnecessary tool calls, information loss, redundant statements, and poor question quality. These combine into a conversation-progression score. Because these measures are judged from the dialog, they're reported for both modes.

Simulate a degraded connection

Real callers connect from noisy environments and unreliable networks. These flags degrade the simulated user's audio so you can test how your agent responds:

Combine them to model a worst-case caller:

lk agent simulate audio --scenarios scenarios.yaml --background-noise --packet-loss

Every lk agent simulate option also applies to audio. For example, run a degraded audio pass against an already-running agent:

lk agent simulate audio --scenarios scenarios.yaml --agent-name my-agent --low-quality-microphone

Run in CI

A checked-in scenarios.yaml lets you automate simulations: the same scenarios run on every pull request, and a regression in multi-turn behavior fails the build like a broken unit test.

The CLI supports this directly. It switches to plain, line-by-line output when stdout isn't a terminal or when CI is set, and it exits non-zero if any scenario fails, so the job fails without extra configuration. Add --yes only when a run generates scenarios from source, because the source-upload confirmation can't be answered non-interactively.

GitHub Actions

Authenticate the CLI with LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET from repository secrets. Because the CLI starts your real agent as a local worker, the workflow must also install your agent's dependencies and provide any API keys the agent itself reads:

name: Agent simulations
on:
  pull_request:

jobs:
  simulate:
    runs-on: ubuntu-latest
    env:
      LIVEKIT_URL: ${{ secrets.LIVEKIT_URL }}
      LIVEKIT_API_KEY: ${{ secrets.LIVEKIT_API_KEY }}
      LIVEKIT_API_SECRET: ${{ secrets.LIVEKIT_API_SECRET }}
      # Any keys your agent's own plugins read.
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv sync
      - run: curl -sSL https://get.livekit.io/cli | bash
      - run: lk agent simulate --scenarios scenarios.yaml --concurrency 5

name: Agent simulations
on:
  pull_request:

jobs:
  simulate:
    runs-on: ubuntu-latest
    env:
      LIVEKIT_URL: ${{ secrets.LIVEKIT_URL }}
      LIVEKIT_API_KEY: ${{ secrets.LIVEKIT_API_KEY }}
      LIVEKIT_API_SECRET: ${{ secrets.LIVEKIT_API_SECRET }}
      # Any keys your agent's own plugins read.
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: curl -sSL https://get.livekit.io/cli | bash
      - run: lk agent simulate --scenarios scenarios.yaml agent.ts

Keep the following in mind when configuring a job:

• Set --concurrency deliberately. It's limited by your project's quota, and every parallel job competes for the same inference capacity. Exceeding the quota can cause jobs to fail on rate-limited LLM completions. The CLI detects this failure and prints the concurrency value to use when re-running.

• Keep pull request runs in text mode. Audio runs are slower and more expensive, so they are better suited to a nightly or pre-release schedule than to every push. To learn more, see Audio simulations.

• Pin your scenarios to absolute dates. Otherwise, time-sensitive scenarios can begin failing months later.
• Follow the dashboard link. Use it to view transcripts for any failed scenario.

Export a run

To analyze a finished run outside the CLI, export it as JSON. The export includes the run, its summary, and the exact per-job chat contexts, which is useful for archiving results as a build artifact or comparing behavior between runs:

lk agent simulate --export <run-id> > run.json

To reopen a previous run in the terminal, pass its ID to --view:

lk agent simulate --view <run-id>

Additional resources