ElephantBench

ElephantBench is a closed-book knowledge probe for evaluating whether a language model remembers long-tail facts and whether it recalls the different verified accounts associated with those facts. The benchmark contains 1,094 questions using two formulation types: a named-entity form and a clue-based form.

The dataset is also available on Hugging Face.

The package provides six commands:

• elephantbench-run: query an OpenAI-compatible model without sources or tools;
• elephantbench-judge: assign complete, partial, or failed recall using an LLM judge;
• elephantbench-score: compute the fixed-denominator C/P/F/K metrics;
• elephantbench-analyze: report per-model metrics and greedy cross-model oracle coverage;
• elephantbench-validate: validate dataset structure and IDs;
• elephantbench-construct: tag raw documents, retrieve and classify document pairs, build conflict subgraphs, synthesize paired questions, verify them, and export benchmark records.

Python 3.10 or newer is required. Run the following commands from this directory. The core package has no third-party runtime dependencies.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Reconstructing the benchmark additionally requires the construction dependencies:

python -m pip install -e '.[construction]'

The runner uses the standard non-streaming OpenAI-compatible /chat/completions interface. Only the system instruction and the benchmark question are sent to the target model. Gold answers, source documents, retrieval tools, and browsing tools are never included in the request.

export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="https://api.openai.com/v1"

elephantbench-run \
  --input data/elephantbench.jsonl \
  --output outputs/my-model.jsonl \
  --model my-model \
  --workers 4 \
  --temperature 0 \
  --max-tokens 8192 \
  --resume

Use --header NAME=VALUE for gateways that require an additional non-secret routing header. Do not put API keys or authentication captures inside this repository.

Judge saved responses with any OpenAI-compatible model:

elephantbench-judge \
  --benchmark data/elephantbench.jsonl \
  --responses outputs/my-model.jsonl \
  --output outputs/my-model.judged.jsonl \
  --judge-model my-judge-model \
  --workers 4 \
  --resume

Then compute the paper metrics:

elephantbench-score \
  --benchmark data/elephantbench.jsonl \
  --results outputs/my-model.judged.jsonl \
  --output outputs/my-model.summary.json

The three outcome rates are mutually exclusive and exhaustive:

• C (complete recall): every verified answer is covered without a material contradiction;
• P (partial recall): at least one, but not every, verified answer is covered;
• F (failed recall): no verified answer is covered, the answer materially contradicts the references, or generation/judging failed.

They satisfy C + P + F = 1. Conditional completeness is K = C / (C + P) and measures complete recall among questions for which the model recalls at least one verified answer. Missing result rows and request failures remain in the fixed benchmark denominator and count toward F.

For subgroup and model-complementarity analysis, pass one or more labeled result files:

elephantbench-analyze \
  --benchmark data/elephantbench.jsonl \
  --result model_a=outputs/model-a.judged.jsonl \
  --result model_b=outputs/model-b.judged.jsonl \
  --output outputs/analysis.json

This reports per-model C/P/F/K, followed by a greedy oracle curve that adds the model contributing the largest number of new complete answers.

The construction pipeline turns prefiltered $D_{\mathrm{low}}$ documents into verified, paired benchmark questions:

D_low documents
  -> full-document store + SuperGPQA++ tagging + T-NER extraction
  -> subject-slot preparation -> candidate retrieval -> relation classification
  -> conflict subgraphs -> paired QA synthesis -> full-document validation
  -> independent web verification -> export

Download the prefiltered $D_{\mathrm{low}}$ source corpus from the ElephantBench-Source repository and initialize the construction workspace:

export DATA_SOURCE=data/ElephantBench-Source
export CONSTRUCTION_OUT=outputs/construction

hf download panzs19/ElephantBench-Source \
  --repo-type dataset \
  --local-dir "$DATA_SOURCE"

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  build-store \
  --input "$DATA_SOURCE"

Configure an OpenAI-compatible endpoint for the model-assisted stages:

export OPENAI_BASE_URL=https://example.test/v1
export OPENAI_API_KEY="..."
export OPENAI_MODEL=model-id

Assign a SuperGPQA++ knowledge-point label to each document:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  tag-kp \
  --input "$DATA_SOURCE" \
  --workers 20 \
  --timeout 1200

Run T-NER over the same documents:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  tag-ner \
  --input "$DATA_SOURCE" \
  --device cuda:0

Use --devices cuda:0,cuda:1,... to distribute shards over multiple GPUs, or use --device cpu when CUDA is unavailable. Full-corpus CPU extraction will be much slower.

Join the two tag streams into (knowledge point, subject, slot) anchors:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  --workers 8 \
  prepare

Generate bounded candidates within (knowledge point, subject, slot) groups and across different knowledge points that share a normalized T-NER entity. Frequency filters and per-document Top-K retrieval keep large entity groups tractable.

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  --workers 8 \
  candidates

Read both full documents and classify each pair as support, conflict, or none:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  classify \
  --processes 20 \
  --threads-per-process 20 \
  --timeout 1200

Attach supporting neighbors to both endpoints of each conflict edge:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  subgraphs

Generate one named-entity question and one clue-based question for each retained conflict group:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  synthesize \
  --workers 8

Use an independent LLM pass to read every full document in the retained subgraph and confirm that each synthesized answer is explicitly reported:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  validate-sources \
  --workers 8

After source validation, use Brave Search and the built-in web fetcher to obtain independent evidence for every proposed answer. Set OPENAI_VERIFY_MODEL=model-id only if the endpoint requires explicit model selection.

export BRAVE_SEARCH_API_KEY="..."

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  verify \
  --workers 4

Export two questions for every verified conflict group:

elephantbench-construct \
  --results-dir "$CONSTRUCTION_OUT" \
  export

Every exported conflict group must be reviewed before release. Reviewers confirm that both questions refer to the intended fact, each answer is independently supported by the retained sources and external evidence, the reported accounts are genuinely incompatible, and the questions contain no answer leakage or material ambiguity. Only groups that pass this review are included in the benchmark.

All intermediate files are written to CONSTRUCTION_OUT. Remote stages resume from completed records by default. Run elephantbench-construct <command> --help for concurrency, retry, filtering, and audit options.

ElephantBench is released under the Apache License 2.0.

@article{pan2026elephantbench,
  title={Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge},
  author={Pan, Zhuoshi and Lu, Junru and Qian, Yan and Zhao, H. Vicky and Yin, Di and Sun, Xing},
  journal={arXiv preprint arXiv:2608.28478},
  year={2026}
}