Summary
Both FrontierChallenge HF datasets currently fail their own tools/verify_*.py on a fresh main download, which aborts setup.sh for every new evaluator.
Details
apodex/FrontierChallenge (solve): README.md at main hashes to e99e2400…, but checksums.sha256 still records 81a5032a… (the README as of revision 79818fdd, before the two citation edits 0cad1a9e and 93e1cc68). tools/verify_dataset.py → checksum failures: README.md.apodex/FrontierChallenge-reference (gated): same pattern — README was edited without regenerating its checksums.sha256 entry, so tools/verify_reference_dataset.py fails the same way. (All 97 verifier archives still pass their manifest/registry commitments — only the doc line is stale.)
Impact
scripts/setup.sh hard-fails at verify_dataset_tools, so a by-the-book install cannot complete without manually pinning README to 79818fdd (solve) / regenerating the checksum line (reference).
Suggested fix
Regenerate checksums.sha256 whenever docs change — a tiny CI check on the dataset repos would prevent recurrence.
Summary
Both FrontierChallenge HF datasets currently fail their own
tools/verify_*.pyon a freshmaindownload, which abortssetup.shfor every new evaluator.Details
apodex/FrontierChallenge(solve):README.mdatmainhashes toe99e2400…, butchecksums.sha256still records81a5032a…(the README as of revision79818fdd, before the two citation edits0cad1a9eand93e1cc68).tools/verify_dataset.py→checksum failures: README.md.apodex/FrontierChallenge-reference(gated): same pattern — README was edited without regenerating itschecksums.sha256entry, sotools/verify_reference_dataset.pyfails the same way. (All 97 verifier archives still pass their manifest/registry commitments — only the doc line is stale.)Impact
scripts/setup.shhard-fails atverify_dataset_tools, so a by-the-book install cannot complete without manually pinning README to79818fdd(solve) / regenerating the checksum line (reference).Suggested fix
Regenerate
checksums.sha256whenever docs change — a tiny CI check on the dataset repos would prevent recurrence.