ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training

Vision-Language-Action (VLA) models have become a central paradigm for generalist robot control. Early large-scale policies such as RT-1 and RT-2 [19, 31] showed that heterogeneous robot demonstrations can be absorbed into a unified language-conditioned policy. More recent systems, including OpenVLA, $\pi_{0}$, $\pi_{0.5}$, GR00T, and Gemini Robotics [2, 4, 5, 3, 1], extend this recipe with pretrained vision-language backbones, diffusion or flow-based action heads, and broader cross-embodiment robot data. By inheriting web-scale visual and semantic priors, these models achieve strong instruction following, object-level generalization, and transfer across diverse manipulation settings.

A complementary line of work uses foundation models as high-level planners rather than end-to-end visuomotor policies. These modular systems decompose robot behavior into semantic reasoning, affordance prediction, task planning, and low-level skill execution [32, 33, 34, 35]. Such decomposition improves interpretability and can reuse existing controllers, but it often depends on predefined skill libraries and carefully engineered interfaces between abstract reasoning and physical execution. End-to-end VLAs reduce this hand-engineered modularity by directly mapping observations and language instructions to actions, but they place a heavier burden on action-labeled robot demonstrations to teach spatial, temporal, and physical regularities.

Despite this progress, most VLA models remain primarily action-centric. Their pretrained backbones are typically optimized on static image-text or video-language objectives, while downstream control is learned through supervised imitation over comparatively limited robot trajectories. As a result, VLAs may inherit rich semantic knowledge about what to do, but often lack an explicit model of how the physical scene evolves under actions. This limitation becomes especially salient in long-horizon manipulation, novel physical skills, contact-rich interaction, and settings where historical context is required to disambiguate the current state. These observations motivate recent efforts to complement VLA-style semantic priors with predictive world-modeling objectives that directly capture temporal dynamics.

World models learn predictive representations of environment dynamics and have long been studied for model-based control and planning [36, 37, 38]. In robotics, they differ mainly in the state representation used for prediction. Latent-space methods learn compact dynamics models for planning or reinforcement learning; 3D methods predict geometric evolution with point clouds or particles; and pixel- or video-space methods directly predict future visual observations [39, 40, 41, 42, 43]. Video-based models are particularly attractive for generalist manipulation because they provide dense supervision from every frame transition and can exploit large-scale video pretraining to capture broad spatiotemporal priors.

Recent work has begun to integrate video prediction with robot policy learning. One family follows an imagine-then-act paradigm: the model first generates future visual states and then recovers actions through inverse dynamics, planning, or a separate policy conditioned on the predicted future [42, 43, 44]. While intuitive, this pipeline can suffer from open-loop drift, mismatch between generated and real observations, and additional latency from test-time video generation. A second family jointly models visual dynamics and actions within a single generative architecture, giving rise to World Action Models (WAMs) [10, 12, 13, 14]. By coupling future-frame prediction with action generation, WAMs use visual dynamics as an auxiliary or implicit planning signal for visuomotor control.

Several recent WAMs instantiate this idea at scale. DreamZero [9] frames WAMs as zero-shot policies by adapting pretrained video diffusion models to jointly predict future videos and actions, showing that video-derived physical priors can improve generalization to unseen tasks and new embodiments. LingBot-VA [11] formulates robot control as autoregressive video-action diffusion, interleaving video and action tokens under causal attention and using KV-cache-based history to preserve long-horizon context. LingBot-VA 2.0 [12] further advocates native video-action pretraining for robot control, introducing an embodiment-oriented visual-action tokenizer, causal pretraining, and sparse expert capacity rather than simply adapting a generic video generator. Together, these works suggest that world modeling is not merely an auxiliary representation-learning objective, but can serve as an independent foundation for robot policy learning alongside vision-language pretraining.

However, existing WAMs still face important deployment challenges. First, high-quality video generation is computationally expensive, and naively placing video prediction on the closed-loop control path can reduce control frequency. Second, many joint world-action architectures bind video prediction and action execution to the same short temporal horizon, even though visual world prediction and low-level motor correction naturally operate at different timescales. These limitations motivate architectures that decouple long-horizon predictive reasoning from high-frequency reactive control while preserving information flow between the two.

Dual-system architectures provide a natural way to reconcile deliberative reasoning with real-time control. In robotics, such systems typically combine a slow but expressive pathway for semantic understanding, planning, or predictive modeling with a fast pathway for reactive action execution. Prior work has explored this principle in several forms: hierarchical manipulation systems and skill-level diffusion planners decompose long-horizon tasks into executable subgoals or skills [45, 46]; dual-policy systems combine a generalist model with a lightweight specialist controller [47, 48, 49]; slow-fast sensorimotor policies use fast feedback pathways for contact-rich correction [50]; and asynchronous VLA or action-chunking systems move expensive model inference off the control-critical path [51, 52]. These approaches show that separating slow reasoning from fast execution can improve deployability, especially when the slow model is too expensive to run at the robot control rate.

Recent WAM-based policies extend this principle from semantic planning to predictive world modeling. AHA-WAM [53] explicitly reorganizes WAM inference into a low-frequency video-DiT world planner and a high-frequency action-DiT executor. Its asynchronous horizon-adaptive design reuses long-horizon planner context across multiple action updates, while observation-guided context routing adapts stale planner context to the latest closed-loop observation. This formulation highlights a key insight: the world branch need not generate short-horizon plans at every control step; instead, it can act as a slower predictive substrate that provides reusable context for a faster action policy.

Robot datasets expose incompatible control interfaces, ranging from Cartesian end-effector commands for a single arm to bi-manual systems with torso, base, and dexterous-hand controls. We map these native interfaces into a 100-dimensional semantic state-action space so that every coordinate retains consistent physical meaning across embodiments. State and action use the same slot layout: state describes the current robot configuration, whereas action represents a future control chunk.

Figure 2 shows the Slow-Fast architecture used for low-latency WAM control. The design separates high-capacity world modeling from high-frequency action generation. A Slow DiT (5B) maintains a strong video-centric world model and produces intermediate visual dynamics representations, while a lightweight Fast DiT (0.5B) consumes these representations together with the latest robot observation and state to predict the action used for control.

The Slow branch operates on the video-action interface introduced above. Given RGB observations, proprioceptive state $s_{t}$, and text instruction $\ell$, the system encodes them into visual latent tokens $z$, state tokens, and language embeddings, respectively. During training, the Slow branch additionally ingests noisy future video tokens alongside noisy action tokens. The primary objective of this branch is to model causal visual dynamics: it predicts future video latents while extracting multi-layer DiT key-value (K/V) features that summarize visual transitions. Notably, auxiliary action supervision is attached to the Slow branch not for runtime deployment, but rather to enforce strong semantic alignment between the predicted video dynamics and the underlying action sequences.

The Fast branch serves as the dedicated action generation module for real-time execution. Rather than executing a high-capacity world model at every control step, the Fast DiT reuses the layer-wise video K/V cache from the Slow branch as conditioning guidance. It leverages updated observation latents, current state tokens, language conditions, and noisy action tokens to predict the final action sequence. This design allows the controller to immediately react to the latest visual and proprioceptive feedback ahead of the Slow branch, while continuously benefiting from its rich spatial-temporal representation.

Mechanistically, the coupling between the two branches occurs directly within the self-attention layer. The Slow DiT computes self-attention over its unified video, action, and state tokens following the causal masking pattern detailed in Section 4.3.2, simultaneously caching the K/V features derived from its video stream at each layer. In the Fast DiT, action queries attend to their own observation, action, and state representations, while concurrently cross-attending to the cached Slow video tokens. As illustrated in Figure 2, this K/V injection effectively bridges the two branches, transferring visual dynamics without the costly explicit generation of full future video sequences. The complete Slow and Fast attention masks are provided in Appendix C.

By default, we instantiate a shallow Fast tower aligned with the early layers of the Slow DiT. Specifically, the Fast branch cross-attends to the video K/V caches from the first $12$ Slow layers, injecting local visual and motion priors while retaining the low latency required for high-frequency control. During real-robot deployment (Section 6.1), the Fast branch directly generates the final control actions.

ZimaBlue is trained with a three-level data pyramid, as shown in Figure 3, that progressively adapts a pre-trained text-to-video diffusion transformer to embodied intelligence. The curriculum separates two forms of supervision, i.e., video and action, which are typically entangled in robot learning. The proposed training paradigm has three stages. Specifically, Stage I, video pretraining, uses large-scale heterogeneous videos to adapt the visual generative prior to causal embodied dynamics without requiring action annotations. Stage II, video-action mid-training, introduces robot trajectories with synchronized observations, proprioceptive states, language instructions, and actions. Rather than treating action prediction as the final objective at this stage, action supervision grounds the Slow world model in control-relevant dynamics and encourages its visual representations to retain motor-relevant information through a unified cross-embodiment action interface. Stage III, post-training, specializes the model to a target embodiment and introduces the lightweight Fast branch for responsive closed-loop control.

For training efficiency, we optimize only the Slow branch during video pretraining and video-action mid-training. Since these stages cover the largest and most heterogeneous data mixtures, we concentrate the training budget on learning a strong video-centric world model instead of jointly optimizing both towers throughout the full curriculum. By the start of post-training, the Slow representations have already been grounded by action supervision, substantially simplifying the mapping from visual dynamics to executable actions. We therefore introduce Fast only at this stage, where it can efficiently specialize to the target deployment domain on top of the aligned Slow representation.

Across all stages, we use a unified flow-matching objective. During video pretraining, it is applied only to visual latents, training the Slow branch to model causal forward dynamics. During video-action mid-training, the same objective is extended to action chunks, using action prediction as an auxiliary alignment signal that injects motor relevance into the shared visual representation. During post-training, Fast branch consumes these aligned Slow features together with the latest observation and proprioceptive state to produce low-latency actions for closed-loop control.

Video pre-training repurposes generic video generation priors from Wan2.2-TI2V-5B [26] into a causal visual dynamics model for embodied interaction. By learning from large-scale egocentric videos without action annotations, this stage acquires generalizable priors over objects, scenes, motion, and contact that provide a strong representational basis for subsequent action grounding. We elaborate on our recipe from two perspectives: data and training strategy.

Building upon the pre-trained backbone, mid-training incorporates real-robot trajectories using the action encoders and decoders detailed in Section 3. Here, the Slow world model jointly predicts future video latents and action chunks. This joint prediction forces action and video tokens to interact within the shared backbone, grounding pre-trained visual dynamics with cross-embodiment action semantics and producing representations rich in motor-relevant information. Note that during this stage, the Fast branch remains inactive.

Post-training specializes the model to a target embodiment by adapting it to the deployment environment, control interface, and task distribution, while preserving the visual dynamics priors acquired during pretraining and the cross-embodiment grounding learned during mid-training. It proceeds in two phases: first, specializing the Slow branch to the target domain, and then training the Fast action branch with Slow frozen.

ZimaBlue accelerates closed-loop control through three complementary mechanisms: asynchronous Slow-Fast inference, diffusion step distillation, and Torch compile. At deployment, the Slow and Fast branches operate at different frequencies: Slow updates long-horizon world-model guidance asynchronously, while Fast uses the latest observation and available Slow guidance to generate high-frequency actions without waiting for each Slow rollout. Following post-training, Distribution Matching Distillation (DMD) [62] is applied sequentially to the Slow branch’s joint video-action generation and the Fast branch’s action generation, reducing each branch from eight to two DiT evaluations.

Figure 4 depicts the asynchronous inference schedule of the Slow-Fast dual system. The design decouples two processes with different temporal requirements: Slow updates long-horizon guidance, while Fast performs high-frequency closed-loop action generation. Instead of waiting for Slow inference to complete, the two streams run concurrently, allowing Fast to continuously refine actions using the latest observations while asynchronously receiving updated Slow guidance.

Following target-domain post-training, we apply DMD [62] as a sequential two-stage distillation procedure. We first distill the task-specialized Slow branch from eight to two DiT evaluations per Slow rollout. The resulting distilled Slow branch retains the joint prediction of future video latents and action tokens. We then freeze the distilled Slow branch and distill the Fast action branch from eight to two DiT evaluations per Fast request, conditioned on the layer-wise video K/V guidance from the distilled Slow branch. The resulting distilled Slow–Fast dual system uses two DiT evaluations per invocation of each branch while preserving the original model architecture, action horizon, and asynchronous closed-loop execution protocol.

For each distillation stage, indexed by $b\in\{S,F\}$ for the Slow and Fast branches, respectively, DMD maintains three networks: a deployable student generator $G_{\theta}^{b}$, a frozen real-score model $D_{r}^{b}$ initialized from the corresponding teacher, and an online fake-score model $D_{f}^{b}$ that tracks the evolving student distribution. The student produces a clean sample $x_{g}^{b}$ from a truncated two-step rollout. For the Slow branch, $x_{g}^{S}=(z_{g},a_{g})$ contains both future-video latents $z_{g}$ and an action chunk $a_{g}$; for the action-only Fast branch, $x_{g}^{F}=a_{g}$.

To estimate the distribution mismatch, we sample a noise level $\sigma\sim\mathcal{U}(0.02,0.98)$ and re-noise $x_{g}^{b}$. The real- and fake-score models then produce the corresponding clean-sample estimates $\hat{x}_{0,r}^{b}$ and $\hat{x}_{0,f}^{b}$. Here, $\hat{x}_{0,r}^{b}$ represents the estimate induced by the frozen teacher distribution, whereas $\hat{x}_{0,f}^{b}$ represents the estimate induced by the current student distribution. Their normalized difference defines the distribution-matching direction

The operator $\operatorname{mean}_{\mathrm{valid}}(\cdot)$ averages only over valid, non-masked video or action dimensions, and $\epsilon_{\mathrm{num}}$ is a small constant for numerical stability. The DMD direction is treated as a stop-gradient target: gradients propagate through the generated sample $x_{g}^{b}$, but not through the two score-model predictions or the normalization term.

We use a truncated two-step UniPC rollout [63] and randomly sample one of the two denoising steps for gradient computation. All preceding steps are evaluated without gradient, so each update backpropagates through only the selected student step. The corresponding shifted noise level is obtained from the scheduler, avoiding reconstruction from discretized timesteps. In both stages, we adopt the two-time-scale optimization scheme of improved DMD [64], updating the fake-score model five times for every generator update to closely track the evolving distribution.

To validate the individual contributions of egocentric video pre-training and multi-embodiment video-action mid-training, we conduct a series of real-robot evaluations. The evaluation suite consists of 12 zero-shot tasks categorized into two distinct protocols: Standard tasks and Perturbed tasks.

We evaluate ZimaBlue across three representative simulation benchmarks—LIBERO-Plus [27], RoboTwin-2.0 [28], and RoboCasa365 [29]—to comprehensively assess key dimensions of generalist manipulation. Specifically, LIBERO-Plus assesses robustness against observation, instruction, and environmental perturbations; RoboTwin 2.0 focuses on bimanual manipulation under domain randomization; and RoboCasa365 evaluates large-scale performance on both seen and unseen household tasks. For simplicity, we conduct all evaluations exclusively on the Slow System.