POSIX tool surface and memory evals (SPEC-47 A/B, BEAM, xAFS) #1398

Goal

Implement the SPEC-47 POSIX tool surface behind a config flag, and build the eval infrastructure to measure whether it beats the existing rich MCP tools on token usage and accuracy — plus add external memory benchmarks (BEAM, xAFS) so those same evals measure the SPEC-82/SPEC-89 roadmap.

Why now

• SPEC-47 claims 80%+ token savings from POSIX-shaped partial reads; that claim has never been measured against our own tools.
• BEAM (ICLR 2026, arXiv:2510.27246) shows every memory system — long-context, RAG, LIGHT, and SOTA commercial (Exabase M-1 ~60%) — is weakest at contradiction resolution, with temporal reasoning and knowledge update close behind. Those are exactly SPEC-89 and SPEC-82. A per-ability BEAM baseline before those specs land turns them into measurable deltas.
• supermemory's xAFS (CC-BY-4.0) is a surface-agnostic agent-filesystem retrieval eval whose headline metric is tokens per correct answer — the exact question the POSIX A/B asks. Vendor-made, so verify before trusting, but the shape is right.
• Their MemoryBench staging (INGEST → SEARCH → ANSWER → EVALUATE → REPORT, checkpointed) matches our benchmarks/ package stages; prior art, not a dependency.

Design decisions (settled)

1. POSIX tools are off by default, enabled by an enable_posix_tools config flag gated at the MCP composition root. Existing tools unchanged.
2. v1 is read-side only: cat, grep, ls, find, tail, man — thin translations over existing API endpoints per SPEC-47's table. Write verbs wait.
3. v1 ships without section indexing and gets measured anyway; section reads (SPEC-47 component 1) land second so the token delta is a measured result, not an assumption.
4. Eval fairness follows the existing benchmarks/ contract: same tasks, same model, same budget — only the tool surface varies (rich-tools vs posix-tools as the provider axis).
5. BEAM starts at the 100K/500K tiers with raw transcript-as-notes ingestion (cheap baseline) before agent-curated ingestion (the real product story). Expect the raw baseline to be unflattering; the curated delta is the result.

Sub-issues

• POSIX read-side MCP tools behind enable_posix_tools
• BEAM integration in the benchmarks package
• Agent-task eval: rich vs POSIX tool surfaces
• xAFS dataset adapter
• Section indexing and range reads (SPEC-47 component 1)
• bm CLI POSIX verbs

References

• SPEC-47: AGENT-FILESYSTEM (specs project) — tool shapes, flags, S3-facade cut rationale
• SPEC-82 (temporal), SPEC-89 (claims/contradictions) — the abilities BEAM measures externally
• BEAM code/data: https://github.com/mohammadtavakoli78/BEAM (paper's runner); Exabase forked Hindsight's open runner — evaluate both before building ours
• mem0 on BEAM: https://mem0.ai/blog/what-is-beam-memory-benchmark-the-paper-that-shows-1m-context-window-isnt-enough
• LoCoMo cross-system leaderboard: https://benchmarks.mnemoverse.com/
• Existing harness: benchmarks/ package (LoCoMo cats 1–4 headline, fairness contract, retrieval/qa/review/diagnose/rejudge stages)

🤖 Generated with Claude Code

https://claude.ai/code/session_014pmKq6bqCi6Zp6BTHuZjrp