Summary
Two related problems make a stock --track open run fail before any trial starts.
(a) task_098_orca_claisen_thermochemistry is selected on the open track but requires the licensed ORCA runtime.
setup.sh --track open reports ready: 81 tasks, but run_eval.sh --track open immediately fails:
FATAL: 1 task(s) require ORCA, but the licensed
local-only image frontierchallenge/orca-user-local:6.0.1 does not exist.
task_098_orca_claisen_thermochemistry
Either the registry image label for task_098 or the open-track selection looks wrong — an evaluator without ORCA must manually --exclude-task-name task_098_orca_claisen_thermochemistry.
(b) The ORCA preflight scans the whole stage directory, ignoring include/exclude filters.
Even with the exclude flag above, the FATAL persists when the task is still present in .frontierchallenge/eval-stage/open/ from a previous launch; the only workaround is deleting it from the stage dir by hand. The preflight should evaluate the effective task selection.
Environment
Linux x86-64, Docker backend, current main, open-track datasets from HF.
Happy to provide logs or a PR for (b).
Summary
Two related problems make a stock
--track openrun fail before any trial starts.(a) task_098_orca_claisen_thermochemistry is selected on the open track but requires the licensed ORCA runtime.
setup.sh --track openreportsready: 81 tasks, butrun_eval.sh --track openimmediately fails:Either the registry image label for task_098 or the open-track selection looks wrong — an evaluator without ORCA must manually
--exclude-task-name task_098_orca_claisen_thermochemistry.(b) The ORCA preflight scans the whole stage directory, ignoring include/exclude filters.
Even with the exclude flag above, the FATAL persists when the task is still present in
.frontierchallenge/eval-stage/open/from a previous launch; the only workaround is deleting it from the stage dir by hand. The preflight should evaluate the effective task selection.Environment
Linux x86-64, Docker backend, current
main, open-track datasets from HF.Happy to provide logs or a PR for (b).