Skip to content

eval.md

Latest commit

 

History

History
210 lines (164 loc) · 10.3 KB

eval.md

File metadata and controls

210 lines (164 loc) · 10.3 KB
Raw

Evaluation

Documentation index · Benchmark registry and extension reference · FrontierSearchBench evaluation

The evaluation layer under benchmarks/public/ converts public datasets into isolated tasks, runs a selected workflow once per question subprocess, collects text or file outputs, and dispatches the benchmark-specific judge. The standalone FrontierSearchBench source and official scorers remain next to it under benchmarks/frontier_search_bench/. The evaluation layer consumes the framework; the framework never imports it.

Install and configure

uv sync --python 3.12 --extra eval --extra sandbox --extra document-readers
cp .env.example .env

Agent credentials use OPENAI_*. Judge credentials use JUDGE_API_KEY and JUDGE_BASE_URL; judge models are pinned by benchmark code. Search and fetch tools use SERPER_API_KEY and JINA_API_KEY.

FrontierSearchBench does not use JUDGE_MODEL: its external scorer pins its own model slate and takes the JUDGE_* credentials under different names. See FrontierSearchBench evaluation.

Two optional settings pay for themselves on any real run:

  • SUMMARY_LLM_BASE_URL + SUMMARY_LLM_MODEL_NAME let web_fetch condense a fetched page instead of dropping the raw page into context. Unset, extraction falls back to OPENAI_BASE_URL / OPENAI_MODEL, so it still works — it just bills whole scraped pages to your primary model. A small fast model is the right choice here; this is called once per fetch.
  • READDOC_VISION_URL / _MODEL / _KEY enable image reading. Unset, an image yields a placeholder note rather than content, so benchmarks with image questions (OfficeQA) lose those points silently.

The runner sets JUDGE_SESSION once per run and every worker subprocess inherits it, so the whole batch shares one upstream session and the identical grading prompt hits the gateway's prompt cache. Gateways that don't know the header ignore it.

The judge model, and why a run may refuse to start

Each benchmark pins the model its official grader used — that pin is what makes a score comparable with published results. Several pins are OpenRouter-style names, which a plain OpenAI key and most gateways cannot route:

Benchmark Pinned judge model Typically needs JUDGE_MODEL?
browsecomp gpt-4.1-2025-04-14 no
hle_text, superchem_text o3-mini-2025-01-31 no
apex, onemillion_bench gpt-4.1-2025-04-14 (global default) no
officeqa, officeqa_full, gdpval none — deterministic scorers n/a
browsecomp_zh gpt-4o often
deepsearchqa google/gemini-2.5-flash yes
frontier_science_research, frontier_science_olympiad openai/gpt-5 yes
widesearch openai/gpt-4.1 yes
xbench_dr_202510 google/gemini-2.0-flash-001 yes

JUDGE_MODEL overrides every pin. The run logs a warning when it does, because a score graded by a substitute model is no longer comparable with the official grader — that belongs in the run's own output, not in someone's memory.

Every run preflights the judge's model and exits 2 before running a single question if it is unreachable, printing the model name and the gateway's own error. Without that check an unreachable judge is silent: it swallows the transport error, returns NOT_ATTEMPTED for every question, and the run completes reporting 0% accuracy — indistinguishable from a model that answered everything wrong, after however long the run took.

Datasets

Nothing under benchmarks/public/datasets/ is tracked by Git, so most benchmarks need local data before they can run. FrontierSearchBench is the exception: its 41 queries and official scorers are bundled under benchmarks/frontier_search_bench/. Other text benchmarks come from one archive; file benchmarks are downloaded per dataset by the script below.

wget https://huggingface.co/datasets/apodex/Deep-Research-Benchmarks/resolve/main/deep_research_benchmarks_260607.zip
unzip -P 'apodex*()_2026' deep_research_benchmarks_260607.zip
mkdir -p benchmarks/public/datasets
mv benchmarks/datasets/* benchmarks/public/datasets/ && rmdir benchmarks/datasets
rm deep_research_benchmarks_260607.zip

Keep the password single-quoted: it contains shell metacharacters, and an unquoted *() fails or expands against the working directory.

The mv is not optional. The published archive still carries the pre-rename benchmarks/datasets/<DatasetName>/ prefix, so extracting it at the repository root writes to a path the runner no longer reads — every text benchmark would then fail to find its data. If FRONTIER_AGENT_DATASETS_DIR is set, move the extracted directories there instead of into benchmarks/public/datasets/.

The archive covers BrowseComp, BrowseComp-ZH, xbench-DR, SuperChem, FrontierScience, DeepSearchQA, and WideSearch. HLE is the one exception — it is not redistributed here. Accept the license on cais/hle, then place its standardized JSONL at benchmarks/public/datasets/HLE-text/standardized_data.jsonl yourself.

Each dataset unpacks to benchmarks/public/datasets/<DatasetName>/standardized_data.jsonl, where the directory is the key its DatasetConfig declares in benchmarks.public.core.registry.REGISTRY. Set FRONTIER_AGENT_DATASETS_DIR to move that root out of the checkout — worth doing once the file corpora are real, since OfficeQA and GDPval are several GB each.

Run

uv run python -m benchmarks.public.runner.run_subprocess \
  --benchmark browsecomp --pipeline stateful-react-agent --profile default \
  --limit 1 --concurrency 1 --out ./results/smoke

Important options include --runs, --limit, --offset, --no-shuffle, --answer-type, --category, and --fs-mode. If --pipeline is omitted, the dataset's registered default is used. Completed result.json files are resumable. Pipeline IDs are exact registry keys: use stateful-react-agent for the single-agent workflow and agent_team or agent_team_report for team workflows. The former react_base pipeline has been removed and consolidated into stateful-react-agent.

FrontierSearchBench

FrontierSearchBench is scored by an external cross-query scorer that ships with its own ground truth, so it deviates from every other benchmark here in three ways: collection and scoring are separate commands, the collection runner reports null accuracy (judge_method: external_pending) rather than misreporting pending answers as wrong, and a comparable score requires an OS-level read boundary around benchmarks/frontier_search_bench/eval/.

FrontierSearchBench evaluation owns that workflow — the isolation requirement, the collect / export / score commands, the scorer's credentials and options, and scripts/evaluate-frontier-search.sh.

Open-book or closed-book

Each benchmark declares whether it is answerable only from what it provides, and the runner applies that by default — so --benchmark officeqa unbinds web_search / web_fetch / download_file without you asking, and --benchmark browsecomp keeps them. The policy is logged at the top of every run, and it reaches the workflow's tool list rather than being advisory.

Benchmark Default Why
officeqa, officeqa_full closed-book The answer is derived from the Treasury corpus mounted at /inputs.
apex closed-book Its own prompt says to use only the provided files.
gdpval open-book Real-world deliverables; the task does not restrict sources.
browsecomp, frontier_search, onemillion_bench, … open-book Web research is the task.

--no-web and --web override it. Reach for --web on a corpus benchmark only knowing the result is no longer comparable with closed-book reports of it — the gap between the two is itself worth measuring, since it tells you how much of a score came from the corpus and how much from the open web.

File benchmarks

These are not in the text archive from Datasets; download their public source data separately:

uv run python benchmarks/public/scripts/download_datasets.py officeqa gdpval onemillion apex

OfficeQA and APEX are gated and require accepting their Hugging Face terms plus HF_TOKEN. OFFICEQA_DOC_MODE=parsed mounts the parsed Treasury corpus; OFFICEQA_DOC_MODE=raw requires downloading PDFs with --raw-pdfs.

Benchmark Default pipeline Inputs and outputs Scoring
OfficeQA / Full stateful-react-agent Treasury corpus at /inputs Official deterministic numeric/text reward
GDPval stateful-react-agent Reference inputs at /inputs; deliverables from /outputs Deterministic artifact structure validation
APEX stateful-react-agent Safely extracted world plus task overlay LLM rubric score
OneMillion-Bench agent_team Text task, no file mounts Weighted LLM rubric score

GDPval intentionally excludes the original benchmark's agentic pairwise grader. The score here validates that requested deliverables exist and are structurally readable; it is not a quality comparison against the human reference deliverable.

OneMillion-Bench scores the weighted fraction of checklist items an answer hits, against the ONEMILLION_PASS threshold (default 0.5). Changing it changes reported accuracy, so leave it alone unless you mean to report a different bar.

Task filesystem contract

The runner translates standardized row metadata into sandbox metadata:

  • _dataset_root: dataset location for resolving relative source files;
  • _sandbox_mounts: read-only inputs mounted under /inputs;
  • _sys_prompt_addendum: benchmark-specific filesystem instructions;
  • _collect_outputs: collect persistent files from /outputs for grading.

Each question has its own /workspace. World archives are checked for path traversal before extraction. Output collection never treats arbitrary workspace files as final deliverables.

Results and progress

Single runs write tasks/, trials/, results.json, and summary.txt. Multi-run evaluation writes one run_<n>/ directory per seed. Inspect progress with:

uv run python -m benchmarks.public.runner.check_progress ./results/run

See benchmarks/README.md for the registry table and dataset layout used when adding a benchmark.