Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models

This directory contains the code and sanitized agentic artifacts needed to reproduce or audit the main experiments reported in the paper.

The package covers:

• UnitySceneBench 720-sample accept/reject evaluation.
• UnitySceneBench true Unity generation-by-training evaluation.
• Distribution and cross-engine generalization.
• Embodied generalization diagnostics on R2R, Gymnasium MuJoCo, and D4RL Gym-MuJoCo.

Raw prompt traces, interactive development prompts, raw reviewer responses, and reviewer prompt templates are intentionally excluded. The release points to result files and sanitized metric tables instead.

• scripts/unityscenebench/: experiment code for UnitySceneBench 720-sample training/evaluation and true Unity generation evaluation.
• scripts/generalization/: deterministic generalization scripts. The raw reviewer prompt harness is not included; sanitized reviewer-score outputs are provided under agentic_artifacts/generalization/.
• scripts/embodied/: embodied diagnostic scripts for R2R and D4RL/Gymnasium MuJoCo final metrics.
• agentic_artifacts/unityscenebench/: main UnitySceneBench result tables, run summaries, split manifests, GPU/run status, and true-generation result summaries.
• agentic_artifacts/generalization/: sanitized generalization rows and the compact table used by the paper figure.
• agentic_artifacts/embodied/: final embodied metric tables and validation summaries.
• agentic_artifacts/manifests/: manifest-level summaries without raw prompt records.
• schemas/: schemas and release notes for reviewer scores and artifacts.

Checkpoints used by the released scripts are stored under:

<workspace>/Opensource_Reproduction/checkpoints

The checkpoint files are staged locally and are planned for a separate Hugging Face upload rather than direct GitHub storage.

UnitySceneBench 720-sample evaluation and generation audit:

python scripts/unityscenebench/run_author_design_exp1_training_data_scaling_generation.py
python scripts/unityscenebench/run_author_design_exp1_true_generation_by_training.py

Generalization:

bash scripts/generalization/run_generalization_experiment.sh
python scripts/generalization/aggregate_generalization_results.py

Embodied diagnostics:

python scripts/embodied/run_r2r_same_sr_main_metric_20260706.py
python scripts/embodied/run_r2r_same_ours_agentic_matched_20260706.py
python scripts/embodied/run_main_metric_validation_20260706.py
python scripts/embodied/run_d4rl_gym_mujoco_fixed_seed_eval.py

These commands assume the same workspace layout as the original run and access to the required engines, datasets, and checkpoints.

Use these files to verify the numbers used in the paper:

• agentic_artifacts/unityscenebench/main_eval_720/all_seed_results.csv
• agentic_artifacts/unityscenebench/main_eval_720/aggregate.csv
• agentic_artifacts/unityscenebench/main_eval_720/results.json
• agentic_artifacts/unityscenebench/scaling_generation/true_generation_720_all/true_generation_by_training_for_figure.csv
• agentic_artifacts/generalization/paper_generalization_table.csv
• agentic_artifacts/generalization/cross_engine_adaptation_sanitized.csv
• agentic_artifacts/embodied/final_main_metric_total_table_20260706.csv

Python dependencies for the released scripts are listed in requirements.txt. Full reruns additionally require the corresponding engine installs and datasets (see REPRODUCTION_GUIDE.md). The release verifier scripts/verify_release_reproduction.py needs only the Python standard library and supports --skip-checkpoints when the checkpoint bundle is not staged locally.

This package is released under the MIT License (see LICENSE).

This package is designed for result reproduction and audit. It excludes:

• raw interactive prompt traces;
• raw reviewer responses and reviewer prompt templates;
• raw generated prompt-top-k files;
• raw worker JSONL traces;
• local virtual environments and engine build caches;
• large raw engine render outputs unless summarized by released metric tables.