feat(core): add BEAM benchmark integration to the benchmarks package - #1407
Conversation
BEAM (arXiv 2510.27246) joins LoCoMo as a second dataset under the same provider abstraction and fairness contract: path-based dataset adapter with tier selection (100K/500K; 10M excluded), raw transcript-as-notes converter with anti-leakage probe-marker stripping (fail-fast on unknown variants), nugget scoring (0/0.5/1 judge, per-question mean), event ordering via LLM alignment + pure-Python Kendall tau-b, per-ability reporting across all ten abilities with per-answer token accounting, and CLI wiring. Dataset is never vendored; the fetch command is documented. Runner provenance (adapted upstream prompts/math and every deviation) is recorded in docs/benchmarks.md. Tests run fully offline against a hand-written miniature fixture with stubbed LLM runners. Also: pin the benchmarks package ruff lint selection (was implicitly tracking ruff's default, which changed in 0.16), and fix two pre-existing pyright errors in untouched provider tests that blocked the typecheck gate. Closes #1400 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp Signed-off-by: phernandez <paul@basicmachines.co>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aec0ebdc66
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| dataset_id = f"beam-{tier.lower()}" | ||
|
|
||
| groups_dir = output_dir / "groups" | ||
| groups_dir.mkdir(parents=True, exist_ok=True) |
There was a problem hiding this comment.
When convert beam is rerun into an existing output directory after the upstream checkout changes—for example, when a newly added chat_trunecated.json contains fewer sessions—this exist_ok path preserves Markdown files that the new conversion no longer emits. The grouped runner subsequently ingests every remaining document under that group's docs directory, so stale conversations can affect retrieval even though queries.json and conversion.json describe only the current source files. Rebuild the groups tree from an empty or atomic temporary directory, or fail when the output is nonempty.
AGENTS.md reference: benchmarks/AGENTS.md:L8-L11
Useful? React with 👍 / 👎.
| "license_note": BEAM_LICENSE_NOTE, | ||
| "dataset_root": str(dataset_root), | ||
| "converter": {"mode": mode, "max_conversations": max_conversations}, | ||
| "created_at_utc": utc_now_iso(), |
There was a problem hiding this comment.
When identical BEAM inputs are converted at different times, this timestamp changes conversion.json; the documented run recipe passes that file as --dataset-path, and the retrieval CLI hashes the entire file for manifest.json's dataset checksum. Consequently, two runs over byte-identical source data receive different dataset checksums, defeating checksum-based provenance comparisons. Keep the timestamp outside the checksummed input manifest or derive the dataset checksum only from stable source metadata and per-file hashes.
AGENTS.md reference: benchmarks/AGENTS.md:L89-L94
Useful? React with 👍 / 👎.
1 participant
Why
Second slice of the POSIX tool surface + memory evals tracker (#1398). BEAM (arXiv 2510.27246, ICLR 2026) joins LoCoMo as a second dataset so we can baseline current tools per memory ability — Contradiction Resolution and Temporal Reasoning are the external yardsticks for the SPEC-89/SPEC-82 roadmap (every published system is weakest there; ~60% SOTA per Exabase).
Closes #1400.
What changed
datasets/beam.py— path-based dataset adapter for the upstream JSON layout (never vendored; fetch command documented), tier selection (100K/500K; 10M explicitly excluded), knowledge-update record normalization.converters/beam_to_corpus.py— raw transcript-as-notes conversion with anti-leakage stripping of upstream probe-index markers (->> i,jand the real-world->> 2,N/Avariant found in the live 100K data), fail-fast if any marker survives cleaning; sha256-recordedconversion.json. Seam designed so an agent-curated ingestion mode can be added without reshaping the adapter.scoring/beam.py— nugget scoring (upstream judge prompt adapted, provenance cited; 0/0.5/1 per nugget, per-question mean) and Event Ordering (LLM alignment + pure-Python Kendall tau-b,final = tau_norm × f1).docs/benchmarks.mdrecords exactly what was adapted frommohammadtavakoli78/BEAM(MIT code / CC BY-SA 4.0 data) and every deviation: the injected question input the upstream judge prompt references but never provides, the degenerate-tau NaN→0.0 mapping, the concise-answer/event-ordering interaction caveat, and why scipy/json_repair/the upstream judge model were not reused. Published numbers must note the judge differs from the paper's — absolute comparability to the paper's tables is explicitly disclaimed; the cross-provider comparison under our fairness contract is the product.How it was built
Reviewed multi-agent workflow (map package + clone upstream + map ingestion → design → implement → offline fixture tests → verify loop → adversarial review). The review's should-fix — the
N/Amarker variant leaking through the anti-leakage strip, caught by checking the live upstream data rather than the fixture — was fixed with the widened pattern plus fail-fast, and two documentation-honesty nits (a falsely-justified publicized helper, the "replicated exactly" tau claim) were cleaned up before commit.Verification
uv run ruff check/ruff format --check— cleanuv run pyright— 0 errorstests/+test-int/) — all passing, fully offline (stubbed LLM runners, hand-written miniature BEAM fixture covering Information Extraction, Knowledge Update, and Event Ordering incl. the marker variants)Notes
🤖 Generated with Claude Code
https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp