IMPACT: Attention Is the Interaction Map for
Scalable Interaction-Aware World Model Training

Within each DiT cross-attention layer, visual queries attend to the text-token sequence. For block $l$, attention head $h$, visual position $p$, and text-token position $j$, the cross-attention probability is

where $d$ is the key dimension. We sum the probability mass assigned to the object-token set $\mathcal{O}$ and average it over the selected blocks $\mathcal{B}$ and attention heads $\mathcal{H}$:

The resulting map $\mathbf{A}\in[0,1]^{T\times H\times W}$ forms an object-conditioned proposal distribution over the video latent. Because it is obtained from the standard conditional forward pass, constructing $\mathbf{A}$ requires no additional visual model or spatial annotation.

Instead of using a single deterministic region, ADS samples $K$ spatially coherent candidates around the attention distribution. We first detach and temper the attention map:

where $\operatorname{sg}[\cdot]$ denotes stop-gradient and $\kappa$ controls the concentration of the proposal distribution. We then transform it into logits:

For each candidate $k\in\{1,\ldots,K\}$, we sample Gaussian noise on a coarse spatiotemporal grid and trilinearly interpolate it to the resolution of $\mathbf{A}$, producing a smooth perturbation field $\boldsymbol{\eta}_{k}$. A soft candidate is generated as

where $\sigma$ controls the perturbation magnitude and $\tau$ is the sampling temperature. Sampling noise on a coarse grid produces coherent spatial variations rather than independent token-wise perturbations.

To prevent candidate scores from being dominated by differences in region size, every candidate is converted into a fixed-area binary region. Specifically, we retain the largest $\rho|\Omega|$ values of $\mathbf{S}_{k}$:

yielding $\mathbf{M}_{k}\in\{0,1\}^{T\times H\times W}$. All candidates consequently cover the same number of latent positions and remain directly comparable.

ADS evaluates every candidate using the mean local prediction error within that region:

A higher value of $d_{k}$ indicates that the candidate covers content that is currently more difficult for the model to predict. The candidate errors are converted into normalized weights:

where $\beta$ controls the concentration of the weighting distribution. The candidate regions are then weighted to form the interaction map

Through this process, the object-conditioned attention determines where candidate regions are sampled, while the current local prediction error determines their relative contribution to $\mathbf{G}$. The entire ADS pipeline is computed without gradient tracking, preventing the model from directly modifying the proposal distribution or candidate weights to reduce the regional objective.

In our implementation, we use $K=8$, an attention tempering exponent $\kappa=0.65$, perturbation scale $\sigma=0.75$, sampling temperature $\tau=1$, and a coarse noise grid of size $5\times 8\times 12$. Each candidate retains $\rho=0.10$ of the spatiotemporal latent positions, and candidate errors are converted into weights using $\beta=0.7$.

The interaction map is derived from cross-attention, creating a dependency between region estimation and the objective guided by that region. Although $\mathbf{G}$ is detached, allowing $\mathcal{L}_{\mathrm{IWS}}$ to update the attention-producing parameters could still cause the cross-attention representation itself to adapt to the regional objective in subsequent training steps. We therefore separate the DiT parameters into the cross-attention parameters $\theta_{\mathrm{ca}}$ and all remaining parameters $\theta_{\mathrm{rest}}$. Their gradients are routed according to

and

In practice, this routing is implemented through two backward passes with parameter-group gradient hooks. The global backward pass retains gradients only for $\theta_{\mathrm{ca}}$, while the IWS backward pass retains gradients only for $\theta_{\mathrm{rest}}$. Consequently, the cross-attention used to estimate interaction regions remains governed by the original uniformly supervised objective, whereas the remaining DiT parameters learn from the spatially reallocated supervision. This separation prevents the regional objective from directly optimizing its own localization signal and decouples interaction-region estimation from interaction-focused optimization.

IMPACT reuses the cross-attention probabilities and token-wise prediction errors already produced during standard world-model training. Its additional computation consists primarily of sampling $K$ candidate masks, evaluating their masked mean errors, and constructing the interaction-weighted objective. No component of ADS or IWS is used at inference time, so the trained world model preserves the original architecture and inference procedure.

We evaluate IMPACT in two manipulation settings to demonstrate the generality of our method across interaction types, applying it to the Wan2.2 and Cosmos-Predict 2.5 backbone in both. For robot-arm manipulation, the model is trained on $\sim$350K 17-frame videos collected from RoboTwin (Chen et al. 2025), conditioned on 14-DoF dual-arm action trajectories, injected through an additional action encoder. For human-hand manipulation, we train on $\sim$256K clips drawn from EgoDex (Hoque et al. 2025) across 118 tasks, using 81-frame videos conditioned on hand-pose videos temporally aligned with the RGB frames, injected through the shared VAE encoder without any additional encoder. Both settings are trained at 720p and optimization is identical across the two settings: we use bf16 mixed precision with FSDP over 8 GPUs, a per-device batch size of 1 with 4-step gradient accumulation (effective global batch size $1\times 8\times 4=32$), a constant learning rate of $2\times 10^{-5}$ after 100 warm-up steps, and we train for one epoch. IMPACT hyperparameters are fixed across both settings: $K=8$, $\sigma=0.75$, coarse grid $5\times 8\times 12$, $\rho=0.10$, $\kappa=0.65$, $\beta=0.7$, $\tau=1$, $\gamma=5$.

For robot-arm manipulation, we evaluate on WorldArena (Shang et al. 2026), which scores dual-arm manipulation along six dimensions: Visual Quality, Motion Quality, Content Consistency, Physics Adherence, 3D Accuracy, and Controllability, spanning 16 normalized metrics, and condenses overall generation quality into a single EWMScore (the mean of the 16 metrics). We report EWMScore together with the six aggregate dimensions in attr /Border [0 0 0] goto name impact.tab:mainTable 1, and provide all 16 metrics in the technical appendix. For human-hand manipulation, we evaluate on the EgoDex (Hoque et al. 2025) test set along two axes (attr /Border [0 0 0] goto name impact.tab:egodexTable 2): visual metrics: FVD (Unterthiner et al. 2018) and FID (Heusel et al. 2017), covering temporal coherence and per-frame appearance quality; and hand-interaction metrics: CLIP-Hand for the local appearance and semantics of the hand and nearby manipulated object, and Hand IoU for the coarse 2D position and scale of the generated hand (Sun et al. 2026).

For robot-arm manipulation, all models follow the WorldArena evaluation settings. Existing baselines comprise general world models: CogVideoX (Yang et al. 2024), Wan 2.6 (Wan Team et al. 2025), and Veo 3.1; embodied world models: GigaWorld-0, Genie Envisioner, Vidar, IRASim (Zhu et al. 2025), and CtrlWorld; and representation-guided models: TesserAct (Zhen et al. 2025), RoboMaster, and WoW. We additionally report two backbone-specific groups to evaluate IMPACT: Wan 2.2-based models include Wan 2.2 (Wan Team et al. 2025), MSE-trained Wan 2.2-AC, and Wan 2.2-AC with IMPACT; Cosmos-based models include Cosmos-Predict 2.5 (text) (NVIDIA et al. 2025), Cosmos-Predict 2.5 (action), and its IMPACT variant. For human-hand manipulation, baselines comprise general video world models: HunyuanVideo-1.5 (Wu et al. 2025) and Cosmos-Predict 2.5 (NVIDIA et al. 2025) and pose-controlled models: MimicMotion (Zhang et al. 2025), MagicPose (Chang et al. 2024), VACE (Jiang et al. 2025), and LOME (Gao et al. 2026).

attr /Border [0 0 0] goto name impact.tab:mainTable 1 shows that IMPACT improves both backbone families. On Cosmos-Predict 2.5 (action), it raises EWMScore from 55.91 to 62.53 (+6.62 points, 11.8%), achieving the best overall result, the best Controllability (71.19), and the second-best Motion Quality (68.22). Interaction Quality increases from 0.5500 to 0.6360 and Action Following from 0.0133 to 0.6260, although the remaining aggregate dimensions decline. On Wan 2.2-AC, IMPACT raises EWMScore from 58.65 to 62.46 (+3.81 points, 6.5%) and improves Visual Quality, Motion Quality, Physics Adherence, 3D Accuracy, and Controllability, attaining the best Physics Adherence (55.87) and 3D Accuracy (92.56) and the second-best Visual Quality (60.60). The best IMPACT result exceeds Wan 2.6, CtrlWorld, and WoW by 0.67, 2.83, and 7.65 points, respectively.

For human-hand manipulation, attr /Border [0 0 0] goto name impact.tab:egodexTable 2 reports that IMPACT attains the best results on visual metrics across both general video world models and pose-controlled models, cutting FVD from 366.12 to 110.94 and FID from 44.71 to 5.79 over the action-conditioned Wan 2.2-AC on the same backbone. It also leads on both hand-interaction metrics, improving CLIP-Hand from 0.921 to 0.952 and Hand IoU from 0.693 to 0.772 over Wan 2.2-AC on the same backbone. These gains indicate stronger local interaction fidelity and hand localization than both the MSE-trained counterpart and the pose-controlled baselines. Together with the robot-arm results, these improvements demonstrate that IMPACT delivers consistent gains across different DiT backbones and control signals.

attr /Border [0 0 0] goto name impact.fig:qualitativeFigure 4 compares generations in robot-arm and human-hand manipulation. In both cases, the baselines follow the instruction loosely and tend to blur or distort the contact region, whether the gripper–object contact for the robot arm or the hand–object contact for the human hand, and some drift away from the target. In contrast, IMPACT produces the specified interaction with a sharper contact region and more coherent object dynamics, while keeping the surrounding scene stable. This clearly demonstrates the effectiveness of IMPACT for interaction-region generation.

attr /Border [0 0 0] goto name impact.fig:ads_calibrationFigure 4 illustrates how ADS calibrates the attention prior in a representative training example. The raw object-conditioned cross-attention map $\mathbf{A}$ is diffuse, spreading across the robot arm, workspace, and background. ADS samples $K{=}8$ candidate regions $\mathbf{M}_{1},\dots,\mathbf{M}_{8}$ from this prior and evaluates them using detached local prediction errors (attr /Border [0 0 0] goto name impact.eq:candidate_scoreEq. 13). Candidates covering the contact region receive higher weights than those dominated by the static background. Their weighted aggregation (attr /Border [0 0 0] goto name impact.eq:interaction_mapEq. 15) produces an interaction map $\mathbf{G}$ that is more concentrated around the interacting arm, gripper, and manipulated object. This example illustrates how ADS refines a coarse attention prior into a more targeted supervision map for IWS.

attr /Border [0 0 0] goto name impact.tab:ads_samplingTable 3 shows the effect of separating the components on WorldArena and reveals their complementarity. Starting from the AC backbone (58.65 EWMScore), IWS provides the more direct gain (+2.89 to 61.54), since it primarily addresses the supervision-allocation mismatch. On top of this, ADS further calibrates the cross-attention prior and delivers an additional improvement (+0.92 to 62.46) over weighting the raw, coarse prior directly. Integrated within a single training step, the two components jointly strengthen interaction-region generation, improving EWMScore by 3.81 points overall.

The robot-arm training corpus comprises simulated bimanual manipulation rollouts from RoboTwin 2.0, generated with the SAPIEN physics engine and the dual-arm Aloha-AgileX embodiment. Each episode provides synchronized RGB observations, camera calibration, robot states, end-effector poses, gripper states, language instructions, and planned action trajectories. We use the head-camera stream at $1280\times 704$ resolution and 10 fps, obtained by sampling every third frame from the 30 fps source video. The synchronized action at each frame is

where $q^{L}$ and $q^{R}$ denote the six joint positions of the left and right arms, and $g^{L}$ and $g^{R}$ denote their gripper coordinates.

The training set contains both clean and domain-randomized rollouts. The randomized regime introduces collision-aware distractors, samples table and background appearances from more than 12,000 textures, varies illumination, perturbs the tabletop height by up to 0.03 m, and displaces the head camera. Clean backgrounds and extreme illumination each occur in 2% of randomized rollouts. The language instructions also vary across equivalent task executions.

The corpus covers 50 tasks spanning pick-and-place, container placement, stacking, object ranking, tool use, switch operation, articulated-object manipulation, bimanual handover, and coordinated dual-arm manipulation. The task distribution is non-uniform and ranges from 99 to 28,117 training windows per task. The four largest tasks are color-based block ranking (28,117 windows), size-based block ranking (27,170), bottle placement into a container (16,881), and block handover (16,723).

Each training example contains 17 consecutive frames with stride one and the corresponding 17 action vectors. Boundary examples are temporally resampled or padded to preserve this fixed length. For action dimension $j$, we normalize using the first and ninety-ninth dataset percentiles, $p_{01,j}$ and $p_{99,j}$:

RGB frames and actions always share the same temporal indices. Spatial bucket sampling and matched crop-and-resize augmentation produce the 720p training inputs.

The human-hand training corpus contains approximately 256K egocentric clips from EgoDex spanning 118 tasks. Each clip is paired with camera intrinsics, per-frame camera poses, articulated hand transforms and a language instruction. Each RGB clip is paired with a rendered hand-pose video constructed from the capture-time 3D hand tracks.

The tasks cover tabletop setup and cleanup, pick-and-place, stacking, assembly and disassembly, folding and wrapping, cooking, washing, device insertion and removal, writing, drawing, and tool use.

Each source clip is converted into an 81-frame training clip. Clips longer than 81 frames are represented by 81 uniformly spaced indices including both endpoints; clips shorter than 81 frames repeat the final index. The same index sequence is applied to RGB and hand-pose videos, preserving frame-level correspondence between the target and condition. Source videos are $1920\times 1080$ at 30 fps, and contiguous 81-frame clips therefore span 2.7 seconds. Training uses matched spatial transformations at 720p.

We annotate each training example once with Qwen2.5-0.5B-Instruct. For both RoboTwin and EgoDex, the task instruction is the sole annotation input; no video frame, pose signal, temporal metadata, or external visual annotation is used. The model extracts compact noun phrases for the manipulated objects while preserving discriminative attributes such as color, material, shape, label, or container type. Agents, body parts, cameras, supporting surfaces, backgrounds, and non-interacting scene elements are excluded.

RoboTwin and EgoDex use the same prompt template below, with {instruction} replaced by the task instruction for the current training clip.

System message
You are a precise language annotation assistant for manipulation training. Given only a task instruction, identify the concrete physical objects directly manipulated by the agent. Return only valid JSON, without Markdown, explanations, or code fences. Use compact English noun phrases and preserve discriminative attributes stated in the instruction, including color, material, shape, printed label, object part, and container type. Exclude the robot, hands, fingers, arms, people, cameras, frames, tables, workspaces, backgrounds, and generic scene regions unless the instruction explicitly identifies one of them as the manipulated object. Do not infer objects that are not stated in the instruction. User message
Task instruction:
{instruction} Return the following JSON object: {  “interacting_objects”: [ {“object”: “compact manipulated-object phrase”} ] } Include every manipulated object explicitly stated in the instruction. Preserve the order in which the objects appear. If the instruction refers to the same object more than once, return it once. Return JSON only.

The extracted object phrase is aligned with its occurrence in the original task instruction after tokenization. All subword tokens associated with the object phrase are included in the object-token set $\mathcal{O}$. We then aggregate the cross-attention associated with $\mathcal{O}$ across attention heads and transformer layers to obtain the object-conditioned attention map $\mathbf{A}$ used by ADS.

EgoDex provides capture-time 3D hand and finger tracks together with per-frame camera calibration. We use these annotations directly and do not estimate hand pose from the compressed RGB video. Each hand is represented by 21 three-dimensional joints, accompanied by a presence mask. The camera metadata provides a $4\times 4$ world-to-camera transformation and four intrinsic parameters for every frame.

For frame $t$, a homogeneous world point $\bar{\mathbf{p}}$ is transformed by the world-to-camera matrix $\mathbf{E}_{t}$ and projected using the intrinsics $(f_{x},f_{y},c_{x},c_{y})_{t}$:

Points with depth $p_{c}^{z}\leq 0.01$ are excluded. The 21 joints follow the wrist, thumb, index, middle, ring, and little-finger ordering. Each finger is connected to the wrist and rendered as an anti-aliased skeleton on a black background. We draw green edges with width 2, blue joints with radius 3, and fingertips with radius 5. Pose videos preserve the RGB resolution and are encoded at 30 fps using H.264 with the YUV420p pixel format and a constant-rate factor of 23.

RGB and pose videos contain the same number of decoded frames and use the same 81-frame index sequence for every training segment. The two modalities therefore remain aligned by frame index throughout temporal sampling. Identical crop and resize parameters are subsequently applied to both modalities.

We instantiate the robot-arm setting with two action-conditioned diffusion transformer backbones: Wan 2.2 TI2V 5B and Cosmos-Predict 2.5 (action). The Wan 2.2 backbone contains 30 transformer blocks with hidden width $d=3072$, 24 attention heads, and an FFN width of 14,336. It receives a 17-frame RGB sequence together with the first-frame visual condition and the synchronized 14-DoF action trajectory. The second model is initialized from Cosmos-Predict2.5-2B and retains its action-conditioning interface for the same robot-arm setting.

Let $\mathbf{a}\in\mathbb{R}^{17\times 14}$ denote the normalized action trajectory. We flatten the complete trajectory into 238 scalars and process it with two independent multilayer perceptrons of identical topology:

with hidden width $4d=12{,}288$. The first encoder produces a $d$-dimensional vector that is added to the timestep embedding. The second produces $6d=18{,}432$ values, reshaped to $6\times d$, which modulate the six adaptive normalization components in every transformer block. Learned binary condition embeddings distinguish action-conditioned and action-dropped examples. Action information is therefore injected globally through the timestep and adaptive normalization pathways.

For both backbones, IMPACT constructs its attention anchor from the object-token cross-attention produced by the native transformer. ADS samples $K=8$ candidate masks by adding Gaussian perturbations with $\sigma=0.75$ to the logit of the detached attention anchor on a $5\times 8\times 12$ coarse grid. The perturbed maps are trilinearly upsampled to the latent resolution, transformed with sampling temperature $\tau=1$ and anchor power $\kappa=0.65$, and thresholded to retain the top $\rho=10\%$ of positions. Detached regional MSEs score the candidate masks, and centered scores are converted into aggregation weights with coefficient $\beta=0.7$. IWS uses the resulting soft interaction map to assign per-position weights in $[1,\gamma]$, with $\gamma=5$.

The human-hand model uses the same 5B transformer and conditions on an 81-frame hand-pose video. RGB targets, the first-frame reference, and pose frames are encoded by the shared Wan VAE without an additional pose encoder. The VAE produces 48-channel latents with temporal and spatial compression factors of 4, 16, and 16, yielding 21 latent timesteps for an 81-frame sequence.

The first latent timestep retains the RGB reference condition. At subsequent timesteps, the 48-channel reference latent is replaced by the temporally aligned 48-channel pose latent. Four binary mask channels are concatenated to form a 52-channel condition $\mathbf{y}$. Before 3D patch embedding, the model concatenates the 48-channel noisy video latent $\mathbf{x}$ with $\mathbf{y}$, giving 100 input channels:

The action mask applies pose replacement only to conditioned samples. RGB and pose inputs share the same temporal indices and spatial transformation.

name impact.sec:full_worldarena_results xyz

Tables attr /Border [0 0 0] goto name impact.tab:worldarena_full_quality8 and attr /Border [0 0 0] goto name impact.tab:worldarena_full_task8 report all 16 normalized WorldArena metrics for the same models and ordering. The first covers the generation-quality dimensions (visual quality, motion quality, content consistency) and the overall EWMScore; the second covers the task-oriented dimensions (physics adherence, 3D accuracy, controllability).