Compare atomic-agent, Hermes, and OpenClaw on the same local chat model (default campaign: gemma-4-26b-a4b) using the public GAIA validation split. Scoring uses the official GAIA question_scorer normalization (exact match).

• npm run build (eval spawns dist/cli/index.js for atomic-agent).
• Managed llama daemons or ATOMIC_AGENT_EVAL_LLAMA_URL.
• For atomic-agent memory fabric: embedding daemon on port 19092 (started by atomic-agent models start when an embedding model is configured).
• Competitors: hermes and openclaw on PATH (skipped automatically when missing).
• Full GAIA (not smoke): HF_TOKEN + npm run eval:agents:datasets.

cp eval-agents/.env.example eval-agents/.env
# edit tokens / URLs

Unit tests (scoring + parsing, no LLM):

npm run eval:agents:lint
npx vitest run --config eval-agents/vitest.config.ts eval-agents/harness

Smoke (committed fixtures, all agents):

npm run eval:agents:smoke

GAIA validation Level 1 (real dataset):

npm run eval:agents:datasets   # once
npm run eval:agents:level1

Filter agents:

ATOMIC_AGENT_EVAL_AGENTS=atomic-agent npm run eval:agents:smoke

Scorecard:

npm run eval:agents:scorecard

Each run writes eval-agents/reports/run-<ISO>/:

• matrix.csv / matrix.jsonl — per (agent × question) row
• environment.json — pinned model, sampling, git SHA

• Chat model: shared llama-server URL for all agents.
• atomic-agent: memory fabric on (chat + embedding daemons when available).
• Hermes / OpenClaw: stock CLI, local OpenAI-compatible endpoint.
• Prompt: GAIA prefix + FINAL ANSWER: convention; scorer is deterministic.
• Hermetic smoke: datasets/gaia/fixtures/smoke-level1.json (no HF leak).

See docs/PHASE0-COMPATIBILITY.md for research notes and known risks.