Documentation index · Evaluation operator guide

The benchmark tree has two deliberately separate areas:

• public/ contains FrontierAgent's integrations for public benchmarks such as GDPval, HLE, OfficeQA, BrowseComp, and FrontierScience, together with their registry, runner, judges, dataset tooling, and local result directories.
• frontier_search_bench/ is the independently maintained FrontierSearchBench source tree. It is kept at the top level so it remains visible and can be synced to its future standalone open-source repository.

The public evaluation harness runs one question per Python subprocess. Runs are independently reproducible, resumable, and protected from a single hung task by a process-level timeout. Use the evaluation guide for installation, credentials, judge preflight, datasets, and result interpretation. This page owns the benchmark registry and extension contract.

Each dataset registers its default workflow and scoring implementation. Passing --pipeline overrides the default.

The runtime registry in benchmarks.public.core.registry.REGISTRY is authoritative.

# Smoke test
uv run python -m benchmarks.public.runner.run_subprocess \
  --benchmark browsecomp --pipeline stateful-react-agent --profile default \
  --limit 1 --concurrency 1 --out ./results/smoke

# Five resumable runs, each with 30 workers
uv run python -m benchmarks.public.runner.run_subprocess \
  --benchmark browsecomp --pipeline stateful-react-agent --profile default \
  --runs 5 --concurrency 30 --out ./results/browsecomp

Omit --pipeline to use the dataset default. Inspect progress and aggregate accuracy with:

uv run python -m benchmarks.public.runner.check_progress ./results/browsecomp

frontier_search is collected first and scored after all answers are exported; see FrontierSearchBench evaluation for its three-step workflow and its isolation requirement. Collection results intentionally contain null accuracy rather than misreporting pending answers as wrong.

Dataset files live outside Git. The evaluation guide owns the download commands, the gated-dataset requirements, and OFFICEQA_DOC_MODE; this table records the layout each dataset must end up with under the datasets root.

File tasks receive read-only inputs under /inputs, work in /workspace, and persist requested deliverables under /outputs. GDPval's OSS scorer validates deliverable presence and readable structure; it does not run the source project's agentic pairwise quality comparison against human work.

benchmarks/
├── public/                    public benchmark integrations and runtime
│   ├── core/                  registry, task schema, and adapters
│   ├── families/              dataset configurations
│   ├── judges/                deterministic and LLM-based scoring
│   ├── runner/                subprocess runner and progress reporting
│   ├── harbor_agent/          benchmark agent invoked by the runner
│   ├── scripts/               dataset download and standardization
│   ├── datasets/              local source data (gitignored)
│   └── results/               local run artifacts (gitignored)
└── frontier_search_bench/     standalone benchmark source and official scorers

1. Put standardized_data.jsonl and optional attachments in benchmarks/public/datasets/<DatasetName>/, or declare a bundled JSON source.
2. Export CONFIGS: list[tuple[str, DatasetConfig]] from a module under benchmarks/public/families/; family modules are auto-discovered.
3. Reuse/register an inline judge, or set scoring_mode="external" and provide an exporter plus batch-scoring entry point.

Standardized inline-scored rows require task_id, task_question, ground_truth, and answer_type. Bundled external-scored datasets may select JSON-list input, omit ground truth from collection tasks, and provide a default answer type through DatasetConfig.