Standalone, reproducible benchmark suite for comparing Basic Memory against competitor memory systems.

• Deterministic retrieval benchmarks (Recall@5/10, MRR, Precision@5, content-hit, latency)
• Optional LLM-as-judge scoring (Pydantic Evals)
• Public artifacts with provenance and reproducibility metadata
• Clean dependency isolation from the core basic-memory repository

• Providers:
  • bm-local (warm bm mcp stdio session)
  • bm-cloud (optional, credential-gated)
  • mem0-local
  • zep-reference (reference-only in v1)
• Datasets:
  • LoCoMo (primary)
  • LongMemEval scaffold (placeholder)
  • Built-in synthetic smoke corpus

uv sync --group dev

Optional judge dependencies:

uv sync --group dev --extra judge

uv run bm-bench datasets fetch --dataset locomo

uv run bm-bench convert locomo

uv run bm-bench run retrieval \
  --providers bm-local,mem0-local \
  --corpus-dir benchmarks/generated/locomo/docs \
  --queries-path benchmarks/generated/locomo/queries.json

uv run bm-bench run judge --run-dir benchmarks/runs/<run-id>

uv run bm-bench publish --run-dir benchmarks/runs/<run-id>

By default this project tracks Basic Memory from main.

Each run manifest stores:

• BM source (github main or local path override)
• resolved BM commit SHA

Local override:

uv run bm-bench run retrieval \
  --bm-local-path /Users/phernandez/dev/basicmachines/basic-memory

mem0-local requires model credentials available in environment.

At minimum, set:

export OPENAI_API_KEY=...

If unavailable, provider status will be recorded as SKIPPED(reason).

bm-local verifies index readiness before querying.

• If the installed bm supports bm status --json, readiness is polled from that output.
• If --json is not available in the installed bm, the benchmark proceeds after reindex.

Per run (benchmarks/runs/<run-id>/):

• manifest.json
• provider-status.json
• per-query-retrieval.jsonl
• retrieval-summary.json
• per-query-judge.jsonl (optional)
• judge-summary.json (optional)
• summary.md

just bench-smoke
just bench-fetch-locomo
just bench-convert-locomo
just bench-run-bm-local
just bench-run-mem0-local
just bench-run-full
just bench-judge
just bench-publish RUN_DIR=benchmarks/runs/<run-id>

Dataset publication follows licensing constraints:

• If redistribution is permitted: snapshot + checksum may be published.
• If not: canonical source links + downloader + checksum verification are published.