Survey and tutorial map

From World Models to World Action Models: A Concise Tutorial for Robotics

Edited: July 25, 2026

A concise tutorial that builds from task-specific worlds and physical AI policies to predictive world models—and finally to models that couple imagined futures with executable robot actions.

Xiaoxiong Zhang, Xiong Zeng, and Wei Zhang

Southern University of Science and Technology; LimX Dynamics

ArXiv GitHub

Architectural view

📘 Introduction

Predicting how the world evolves under actions is central to physical AI and generative simulation. A unified architecture makes clear what a model represents, what it predicts, and how its predictions support controller design, planning, decision making, and policy learning.

World

The world is the set of task-relevant entities, including both the robot and its environment. The environment contains the objects of interest and the ambient environment.

Physical AI Task and Policy

The world state is the joint task-relevant state of the robot, objects, other agents, and surrounding environment. A physical AI task asks a policy to drive the world from an initial state toward a goal set while satisfying task-specific constraints.

Autonomous driving includes the vehicle, traffic participants, and road environment; robotic table-cleaning includes the robot, dishes, rag, table, and household surroundings.

Autonomous driving and robotic table-cleaning as two examples of physical AI tasks. — Figure 2. Physical AI tasks: autonomous driving and robotic table-cleaning.

Policy framework for physical AI tasks

A policy maps human intent I and the current observation o_t to a robot action a_t. It may be realized by a modular decision-planning-control pipeline, a fast-slow architecture, or an end-to-end policy.

Human intent and the current observation feed a policy that sends an action to the robot. — Figure 3. A policy framework for accomplishing physical AI tasks.

World Models and World Action Models

For a specified world, a world model predicts how a future observation o_t+1 or state x_t+1 evolves under action a_t, typically conditioned on observation o_t.

The model can be a symbolic dynamics equation, a neural dynamics model, or a diffusion-based video predictor. Observations may be RGB or RGB-D images, point clouds, or proprioceptive states; predicted states may be object poses, keypoints, latent states, or other task-relevant variables.

A world model predicts future observations or states from observation history and an action. — Figure 4. World (action) models.

A world action model predicts future actions and optionally future observations from the current observation and current action. — Figure 4. World (action) models.

Examples spanning symbolic, neural, and diffusion-based world models. — Figure 5. Examples of world models.

From prediction to action

World Action Model

A world action model is a policy whose action generation is coupled, during training or inference, with a model or representation of future world evolution. This coupling may be explicit, latent, joint, or auxiliary.

Explore the four WAM paradigms →

Predictive modeling choices

🧭 World Model Design Space

We first divide world models into two formulations according to the space in which prediction is performed: observation-space world models and state-space world models.

🔮 Observation-Space World Models

These models predict future observations directly, such as RGB images, RGB-D frames, or point clouds.

🧩 State-Space World Models

These models first abstract observations into a state representation, then predict how that state evolves.

🔮 Observation-Space World Models

Classification criteria: observation explicitness and action abstraction.

For observation-space world models, we classify methods by what future observation they generate and how the conditioning action is represented. This yields a two-axis design space.

Figure 6. Design space of observation-space world models.

🧩 State-Space World Models

Classification criteria: the state representation used for prediction.

For state-space world models, we classify methods by the representation that mediates prediction. The key question is what information is retained from observations before dynamics are modeled.

Figure 7. Design space of state-space world models.

From future prediction to action

🤖 World Action Models

World action models bridge future-oriented visual prediction and physical decision making. Existing work differs in how explicitly it represents future observations and how tightly those futures are coupled with action generation.

Taxonomy of world action model paradigms. — Figure 8. Taxonomy of world action models.

Explicit plan

Imagine-then-execute

Generate visual subgoals or rollouts first, then use inverse dynamics, pose estimation, flow, or goal-conditioned policies to produce executable actions.

Feature transfer

Video-feature-conditioned action prediction

Reuse internal representations from video prediction backbones without decoding full future frames at inference time.

Unified model

Joint video-action modeling

Learn a shared generative distribution over future observations and corresponding robot action sequences.

Training signal

Auxiliary video prediction for policy learning

Use future prediction as an auxiliary objective to shape policy representations, then remove the video branch during deployment.

Survey bibliography

📚 Resource Browser

The browser covers the deduplicated references in the survey and follows the taxonomy used in the paper. Some works are cross-listed when they appear in both world-model and world-action-model sections.

Feedback and updates

💬 Contribute to this survey

If you have suggestions, ideas for improving the taxonomy, works that should be included, or would like to discuss world models and world action models, please contact us.

12433017@mail.sustech.edu.cn

Cite this survey

✍️ Citation

The paper is currently represented by the local LaTeX source and compiled PDF. Update venue and publication metadata here as the manuscript changes.

@article{zhang2026worldactionmodels,
  title   = {From World Models to World Action Models: A Concise Tutorial for Robotics},
  author  = {Zhang, Xiaoxiong and Zeng, Xiong and Zhang, Wei},
  year    = {2026},
  note    = {Survey manuscript}
}