BEAM benchmark integration in the benchmarks package #1400

Part of the POSIX tool surface + memory evals tracker.

Scope

Add BEAM (arXiv:2510.27246, code/data) as a second dataset in the benchmarks/ package beside LoCoMo, under the same provider abstraction and fairness contract.

Plan

1. Evaluate existing runners first: the paper's repo and the Hindsight open-source runner that Exabase forked. Reuse before building.
2. Tiers: 100K and 500K conversations only in v1 (1M/10M are a cost decision for later).
3. Ingestion mode v1: raw transcript-as-notes — conversation turns land as BM notes in a project, retrieval answers the probes. Agent-curated ingestion (notes written through the canonical write path by an agent) is a follow-up mode; the delta between the two is the product-relevant result.
4. Scoring: BEAM's nugget methodology (atomic criteria, LLM judge, 0/0.5/1; Kendall tau-b for event ordering). Reuse the package's qa/review/rejudge stage machinery where it fits.
5. Reporting: per-ability scores (Abstention, Contradiction Resolution, Event Ordering, Information Extraction, Instruction Following, Knowledge Update, Multi-hop, Preference Following, Summarization, Temporal Reasoning) — not just an average. Contradiction Resolution and Temporal Reasoning are the SPEC-89/SPEC-82 yardsticks.
6. Token accounting: end-to-end tokens per answer (Exabase's framing), comparable across providers.

Acceptance

• bm-bench run beam --tier 100k produces artifacts with per-ability scores and token counts for the BM provider.
• mem0-local comparator runs under the identical fairness contract.
• Baseline numbers recorded before SPEC-82/89 work lands.
• Runner provenance documented (which upstream runner, which modifications).

🤖 Generated with Claude Code

https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp