Summary
benchmarks/public/runner/run_subprocess.py passes the parsed open-book flag directly into resolve_closed_book(), whose override parameter means closed_book. The two CLI flags therefore do the opposite of what their help text says.
Repro
uv run python -m benchmarks.public.runner.run_subprocess \
--benchmark frontier_science_olympiad --limit 1 --no-web --out ./results/x
Log output:
Book policy: open-book (--web/--no-web) — web tools available
Passing --web instead yields closed-book … web tools unbound.
Cause
closed = resolve_closed_book(args.benchmark, getattr(args, "web", None))
args.web is True for --web / False for --no-web (an open-book flag), while resolve_closed_book(benchmark, override) returns override as the closed-book decision.
Suggested fix
_web = getattr(args, "web", None)
closed = resolve_closed_book(args.benchmark, None if _web is None else (not _web))
Found while running FrontierScience-Olympiad locally; happy to send a PR.
Summary
benchmarks/public/runner/run_subprocess.pypasses the parsed open-book flag directly intoresolve_closed_book(), whoseoverrideparameter means closed_book. The two CLI flags therefore do the opposite of what their help text says.Repro
Log output:
Passing
--webinstead yieldsclosed-book … web tools unbound.Cause
args.webis True for--web/ False for--no-web(an open-book flag), whileresolve_closed_book(benchmark, override)returnsoverrideas the closed-book decision.Suggested fix
Found while running FrontierScience-Olympiad locally; happy to send a PR.