This is the official project repository for Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators.

TL;DR: We study visual spatial reasoning as active visual evidence acquisition. Astra lets a VLM decide when to query an action-conditioned world simulator, inspect the imagined view, and ground the final answer in both observed and simulated visual evidence.

Spatial reasoning from limited egocentric observations often requires evidence that is not directly visible. Conventional text-oriented chain-of-thought over fixed images provides limited gains in such settings. Astra reframes this problem as thinking with imagination: a policy can request a missing viewpoint from a learned world simulator and use the returned observation as spatial evidence.

The framework contains two main components:

• Astra-VL: an agentic VLM policy and reasoner that decides when to imagine, plans camera-motion queries, and grounds the returned visual evidence before answering.
• Astra-WM: an action-conditioned world simulator that synthesizes in-context novel observations from context images and natural-language camera-motion instructions.

Astra couples an agentic VLM policy with an action-conditioned world simulator, then trains the policy to acquire tools and use imagined observations selectively.

1. Spatial consistency matters. Plausible generation is not enough: only an action-faithful world simulator turns imagined views into reliable spatial evidence and reasoning gains.
2. Simulator use must be learned. Strong proprietary VLMs can benefit from Astra-WM directly, while open-source VLMs need agentic training to decide when and how to imagine.
3. Selective imagination beats tool overuse. Rewarding tool calls alone leads to excessive simulator use, while the two-phase curriculum learns when imagined evidence is actually helpful.
4. Imagination helps when evidence is viewpoint-dependent. Agentic tool use keeps camera-centric gains while avoiding generated views when the original context is already sufficient.

We compare Direct Answer, Forced Tool-Use, and Agentic Tool-Use settings. Values in parentheses denote absolute changes over the corresponding Direct Answer result of the same model.

The same trained Astra policy is evaluated under no-tool/direct-answer, forced-tool, and agentic-tool modes. Agentic tool use preserves the gains on camera-centric relations while avoiding unnecessary simulator calls when the original context is more reliable.

• Astra-VL and Astra-WM evaluation scripts
• Astra-WM checkpoints
• Astra-VL checkpoints
• Astra training code

If you find this project useful, please cite:

@misc{zhu2026thinkingimaginationagenticvisual,
      title={Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators}, 
      author={Chenming Zhu and Jingli Lin and Yilin Long and Peizhou Cao and Tai Wang and Jiangmiao Pang and Xihui Liu},
      year={2026},
      eprint={2606.06476},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.06476}, 
}

If you have any questions, please contact chaimzhu@connect.hku.hk.

We sincerely appreciate the following projects for their valuable codebase and benchmark: Verl, vllm-omni, SenseNova-MARS, MMSI-Bench, MindCube.