RestoreBench

RestoreBench is a benchmark for restoring AC power-flow convergence with LLM agents.

Every scenario is a power-grid snapshot for which the AC power flow does not converge under the benchmark solver settings. The agent interprets the grid model and sequentially proposes reactive-control maneuvers, receiving solver-grounded feedback after each action. The episode is successful if AC convergence is restored and the resulting state satisfies all feasibility constraints within 10 maneuvers.

non-convergent snapshot
  → solver-grounded Q–V diagnostics
  → agent-selected atomic maneuver
  → isolated pandapower verification
  → next state, or terminal result

pandapower owns every electrical result and the success verdict. The agent interprets evidence and chooses maneuvers; it never decides whether it won.

Non-convergence is the numerical face of voltage instability: restoring a solvable operating point with a handful of reactive controls is a real operator task, and it makes a sharp testbed for LLM agents — every proposed action is checked by a physics solver, so ungrounded reasoning has nowhere to hide.

• Solvable scenarios. Every scenario includes a deterministic witness maneuver sequence that restores AC power-flow convergence within the allowed action space and budget.
• Solver-grounded evaluation. Success is determined exclusively by the solver, under fixed settings and feasibility constraints. A converged state that violates generator-P/Q or external-grid limits, or that leaves part of the network de-energized, is not considered successful. Bus-voltage limits are deliberately excluded from the verdict and reported separately as solution quality.
• Fixed interaction budget. Each episode allows at most 10 maneuvers. Invalid, malformed, or inapplicable actions consume one maneuver without modifying the network state.
• Consistent evaluation across grids. All scenarios use the same action space, solver policy, feasibility criteria, and evaluation protocol.
• Frozen and verified datasets. Dataset artifacts are versioned and checked against their manifests, while scenarios and witnesses can be independently replayed with restorebench-verify.
• Versioned results. Each result records the dataset, solver, action-policy, ranking-policy, and result-schema versions used to produce it. Incompatible results cannot be aggregated.
• Model-independent scoring. Maneuver sequences can be generated by any model or agent scaffold and evaluated independently with the standalone scorer.
• Reference agent configurations. The repository includes chatbot, single-agent, and multi-agent configurations to compare the effect of tool access and agent architecture.

Input. Scenario Card of a non-convergent snapshot: topology, network state, operating limits and available actions

Action space. Three atomic maneuvers, one per step:

Active-power redispatch, topology switching, load control, and component insertion are out of scope: the benchmark isolates reactive control.

Success. AC convergence under locked solver settings (init="dc", max_iteration=30) within a budget of ten maneuvers, in a state that also satisfies the declared non-voltage feasibility constraints — generator P/Q limits, external-grid P/Q bounds, and connectivity. Bus-voltage quality is measured and reported separately, never folded into the verdict.

Two frozen corpora ship with the repository, under dataset/:

Every shipped scenario is an evaluation scenario. Both corpora were curated out of much larger generated pools; selection is grouped by leakage group, so no two shipped scenarios share a generation recipe with each other's provenance.

Each corpus is laid out the same way:

dataset/<grid>/
    <GRID>_BASE_CASE.json       the unmodified network
    <GRID>_AUGMENTED.json       the healthy augmented network the corpus was built from
    full/S####.json             the scenario snapshot, ground truth for scoring
    lean/S####.json             the reduced snapshot handed to the runtime
    llm/S####.md                the Scenario Card the agent reads
    private/labels.json         curation labels: regime, witness length, boundary evidence
    private/witnesses.json      a known resolving maneuver sequence per scenario
    manifest.json               artifact hashes and policy versions
    evaluation_manifest.json    the scenario roster the loader validates against

Python 3.11, with a uv lockfile pinning every transitive dependency.

1. ⭐ Star the repository on GitHub to support the project!

2. Clone and sync:

   git clone <repository-url>
   cd RestoreBench
   uv sync                   # installs the package and its console entry points

3. Check the install:

   uv run pytest -q                  # full suite, ~15 min
   uv run pytest -q -m "not slow"    # skips the full-corpus checks, ~8 min

The scorer is standalone. Produce a maneuver sequence however you like — any model, any scaffold — write it to JSON, and the scorer replays it against the frozen snapshot:

{
  "scenario_id": "S0008",
  "maneuvers": [
    {"type": "GEN_V_SETPOINT", "gen_id": 33, "new_vm_pu": 0.96}
  ]
}

uv run restorebench-score attempt.json
uv run restorebench-score --batch attempts/ --out results/my_run

The report gives the status (SUCCESS / BUDGET_EXHAUSTED), the per-step outcome of every maneuver, the number of invalid actions, and the quality of the terminal state. A batch report additionally aggregates success rate and maneuver counts, broken down by dataset split.

To sanity-check the setup, score a scenario's own witness from private/witnesses.json — it must come back SUCCESS.

Three configurations are included, differing in whether the agent has physics tools and in how the work is split across roles:

The exact prompts used in the experiments ship with the implementation: system and user templates per configuration in restorebench/agents/ (baseline_chatbot.py, single_agent.py, analyst.py, executor.py), the shared fragments — action vocabulary, cause taxonomy, composition rules — in restorebench/agents/prompt_fragments.py, the tool descriptions and schemas in restorebench/agents/tool_loop.py, and the runtime substitutions (Scenario Card, diagnostics, maneuver history, failure feedback) in their renderers under restorebench/environment/ and restorebench/agents/history_render.py.

To re-run a published experiment, use the campaign that defines it. Everything the sweep needs — the exact scenarios, models, configurations, repetitions and wall-clock limit — is frozen in restorebench/sweeps/campaigns.json, so a provider credential is the only thing you have to supply:

uv run restorebench-sweep --campaign ieee118-anthropic --dry-run   # print the queue, spend nothing
uv run restorebench-sweep --campaign ieee118-anthropic             # 46 scenarios × 9 cells
uv run restorebench-sweep --campaign ieee118-bedrock
uv run restorebench-sweep --campaign ieee118-openai
uv run restorebench-sweep --campaign pegase89-anthropic
uv run restorebench-sweep --campaign pegase89-bedrock
uv run restorebench-sweep --campaign pegase89-openai

The sweep is resumable and writes one scenario at a time, so an interrupted run leaves whole cases rather than half-measured ones. It waits out provider outages instead of recording them as model failures, and stops rather than burning the queue on a dead credential. Re-running it skips cells that are already stored, and prints running spend so you can stop when you have seen enough.

For a one-off run outside a published campaign, drive the harness directly:

uv run restorebench-eval \
    --model claude-opus-5 \
    --configuration 2 --configuration 3 \
    --data-dir dataset/ieee118 \
    --results-dir results/my_sweep \
    --repeats 5

Results are written one JSON per cell (scenario × configuration × model × repetition).

uv run restorebench-generate --network case118 --n 200 --output-dir data/staging/my-corpus
uv run restorebench-validate --dataset-dir data/staging/my-corpus

--network accepts case118 and case89pegase. Generation is long and crash-resumable: pass --checkpoint-root and --resume to pick an interrupted run back up where it stopped.

Turning a result store into plot-ready tables is two steps, both of which read only stored results:

uv run python -m restorebench.sweeps.build_figure_dataset
uv run python -m restorebench.sweeps.build_figure_tables

They write to reports/. No result cells ship with this repository — every number these builders produce is one you measured yourself. The published aggregate figures are reproduced in the Results section below and can be rebuilt from any result store with restorebench-figures.

All seven paper figures are rebuilt from any result store with a single command:

uv sync --group plots
uv run restorebench-figures --store results/ieee118-bedrock --store results/ieee118-anthropic

Every percentage is computed from the stored cells at build time — nothing is hand-entered — and the computed tables are printed next to the rendered PDFs, so each figure is auditable against the store it came from. Episodes whose feedback chain cannot be attributed unambiguously are excluded and reported, never guessed; pooling stores with different corpus or policy version stamps is refused. Figures with no qualifying episodes are skipped with a message rather than rendered empty.

Live provider calls dispatch on model id to one of three transports, all credentialed from the environment and never from code:

The GPT-5.6 Sol experiments are frozen as the ieee118-openai and pegase89-openai campaigns; the transport, tool-call translation and reasoning settings are handled by the model registry:

export OPENAI_API_KEY=sk-...
uv run restorebench-sweep --campaign ieee118-openai
uv run restorebench-sweep --campaign pegase89-openai

Adding a further OpenAI model is a registry edit in restorebench/llm/models.py (id, slug, price) plus, if the model supports it, a reasoning entry in restorebench/llm/openai_provider.py.

The test suite mocks every provider client and needs no network access. Tests that do call a live provider are opt-in behind RESTOREBENCH_LLM_INTEGRATION=1.

restorebench/
    agents/         the agent configurations: prompt templates, tool loops, reprompts
    environment/    resolution runner, scenario loader, Scenario Card rendering
    physics/        feasibility evaluation, action semantics, ranking policy
    tools/          power flow, sandbox, topology — the agent-facing tools
    llm/            provider transports and the model registry
    schemas/        the pydantic contracts every layer validates against
    eval/           the evaluation harness and its result store
    scoring/        the standalone scorer and the one-shot runner
    sweeps/         sweep runners and report builders
    analysis/       result aggregation, stratified metrics, bootstrap CIs
    corpus/         corpus generation pipeline
dataset/            the two frozen corpora
tests/              mirrors the package, one directory per subpackage

results/ and reports/ are run outputs and are not versioned. Seven console entry points are installed with the package:

Every published experiment is re-runnable from a clean clone with nothing but a provider credential: restorebench/sweeps/campaigns.json freezes the scenario list, the model suite, the configurations, the repetitions and the wall-clock limit, and restorebench-sweep runs them.

The corpus itself can be checked without any credential at all:

uv run restorebench-verify --dataset-dir dataset/pegase89

For every scenario this re-derives the benchmark's central claim: the artifacts hash to what the manifest says, the snapshot genuinely does not converge, and the shipped witness resolves it within budget. A failed scenario is therefore a model failure, never an impossible instance.

Results carry a version stamp — dataset version, solver version, action policy, ranking policy, and result schema. Aggregation refuses to compare results whose stamps disagree, and refuses to place an unstamped result in a benchmark table. This is enforced in code, not by convention: an incomparable result raises rather than quietly averaging into a number.

Published results of the benchmark: seven LLMs (GPT-5.6 Sol, Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5, DeepSeek V3.2, Kimi K2.5, GLM-5) across the three interaction architectures, on both grids. Every figure below is built by restorebench-figures from the stored result cells; vector PDFs of each figure live next to the PNGs in docs/figs/results/.

Success rate

IEEE 118	PEGASE 89

Cost against success

IEEE 118	PEGASE 89

Voltage quality and runtime

IEEE 118	PEGASE 89

Failure composition

IEEE 118	PEGASE 89

Maneuver progress

IEEE 118	PEGASE 89

Timeouts

IEEE 118	PEGASE 89

Work in progress — a paper describing this benchmark is under review; citation details will be added here after the review process. The base networks carry their own citation requirements — see below.

The code is MIT licensed. The corpora under dataset/ are released separately under CC BY 4.0 — see dataset/LICENSE.

Neither base network is original to this project. Both come from pandapower (BSD-3-Clause), which sources them from MATPOWER: the standard IEEE 118-bus test case, and the PEGASE 89-bus case from the PEGASE project (Josz et al. 2016; Fliscounakis et al. 2013). The pipeline in restorebench/corpus/ augments and stresses them; the base networks themselves are unmodified inputs. If you redistribute the corpora, carry the citations in dataset/LICENSE with them.