Skip to content

--no-web/--web are inverted by the public benchmark runner #26

Description

@wayfind

Summary

benchmarks/public/runner/run_subprocess.py passes the parsed open-book flag directly into resolve_closed_book(), whose override parameter means closed_book. The two CLI flags therefore do the opposite of what their help text says.

Repro

uv run python -m benchmarks.public.runner.run_subprocess \
  --benchmark frontier_science_olympiad --limit 1 --no-web --out ./results/x

Log output:

Book policy: open-book (--web/--no-web) — web tools available

Passing --web instead yields closed-book … web tools unbound.

Cause

closed = resolve_closed_book(args.benchmark, getattr(args, "web", None))

args.web is True for --web / False for --no-web (an open-book flag), while resolve_closed_book(benchmark, override) returns override as the closed-book decision.

Suggested fix

_web = getattr(args, "web", None)
closed = resolve_closed_book(args.benchmark, None if _web is None else (not _web))

Found while running FrontierScience-Olympiad locally; happy to send a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions