Video Models as Native 4D Renderers:
World-Grounded Conditioning from Animated Mesh

Bringing an animated shot to the screen has traditionally meant committing to a full production pipeline, whose form depends on the medium. 2D animation composes hand-drawn or vector keyframes that are inbetweened and composited, and recent methods learn to synthesize such vector animations directly (Chen et al., 2026c). 3D and CG animation for film and games instead builds an explicit world: assets are modeled, generated from multimodal or compositional inputs (Chen et al., 2025a; Weng et al., 2026a), and edited (Weng et al., 2026b), then rigged and driven by learned skeleton and motion generation (Sun et al., 2025; Sun et al., 2026), and finally textured, lit, and rendered with extensive support from texture and material synthesis and physically based (neural) rendering (Chen et al., 2023; Gao et al., 2024; Zeng et al., 2024; Liang et al., 2025; Xue et al., 2025). Scene-level animation scales this to whole environments that must stay consistent across views and over time (Chen et al., 2024a; Huang et al., 2025; Höllein et al., 2024), while video generation animates directly in pixel space through pose- and skeleton-conditioned character synthesis (Xu et al., 2024; Chen et al., 2026a) and general controllable generators (Wan et al., 2025; Jiang et al., 2025). These pipelines reach high visual quality, but they sit at two extremes: classical CG needs a fully specified renderable scene, whereas pure video generation leaves the underlying 3D/4D state implicit and hard to control precisely.

This gap motivates generative rendering: rather than authoring materials and lighting and invoking a classical renderer, one specifies only the scene geometry and lets a pretrained video diffusion model act as the renderer (Cai et al., 2024; Gu et al., 2025; Kim et al., 2025; Huang et al., 2026; Zhang et al., 2025). The open question is which geometric condition to expose. Camera-control models steer viewpoint through ray or pose signals (Wan et al., 2025; Bai et al., 2025), depth-, edge-, and normal-conditioned systems use projected geometry for layout (ByteDance Seed Team, 2025; Alhaija et al., 2025), and 3D-conditional or tracking-guided methods render from coarse geometry, tracked points, or explicit 3D inputs (Chen et al., 2026f). Their control is typically single-axis, however: camera controls ignore the animated surface state, depth is view-dependent, and static anchors do not handle articulated motion. A parallel lineage shows that dense coordinate maps ease cross-view reasoning, from functional maps, DensePose, and NOCS to UV-space diffusion and neural G-buffers (Ovsjanikov et al., 2012; Guler et al., 2018; Wang et al., 2019; Feng et al., 2018; Miao et al., 2026; Yu et al., 2023; Chen et al., 2024b). DAR unifies these views: it keeps Wan2.2’s camera prior but conditions camera rays and a per-frame 4D G-buffer together, giving one control signal for both observer and object motion.

Training and evaluating a video renderer requires paired dynamic data, and a growing set of resources supplies parts of it. Large static-object and scene collections provide geometry and appearance diversity (Deitke et al., 2023; Ling et al., 2024), synthetic engines and generative domain randomization render controllable data with ground-truth buffers (Greff et al., 2022; Geng et al., 2025), and non-rigid, dynamic-stereo, and 4D or physics-oriented corpora add deformable and time-varying supervision (Li et al., 2021; Karaev et al., 2023; Wen et al., 2025; Zhou et al., 2026; Wu et al., 2025). A recent line turns monocular or multi-view video into 4D or world-scale scenes (Chen et al., 2026e; Chen et al., 2026b; Chen et al., 2025b; Yang et al., 2026), which is exactly the kind of pipeline that yields paired video and 4D-scene data. DAR-4D is complementary to these resources: rather than proposing new capture or reconstruction, it converts heterogeneous assets into a single renderer-ready format with paired RGB, calibrated cameras, Plücker rays, and per-frame depth, normal, world-position, and tracking maps, so that alternative conditioning representations can be compared under matched renderer inputs. Appendix A.2 details the dataset landscape.

Let $\mathcal{S}$ denote the canonical surface domain of an animated mesh. At video time $t$, the animation maps a surface point $s\!\in\!\mathcal{S}$ to world position and normal

A camera $C_{t}=(K_{t},R_{t},o_{t})$ projects visible surface points to pixels. For pixel $u=(x,y)$, let $s_{t}(u)$ be the visible surface point selected by rasterization, if any. The target renderer should produce

where $I_{\rm ref}$ is the first-frame appearance reference and $\tau$ is an optional text prompt. The geometry and camera are not latent variables to be invented by the model; they are user-specified state.

For each pixel $u$ and time $t$, Wan2.2-Control-Camera forms a Plücker ray

where $d_{t}(u)$ is the world-space ray direction and $o_{t}$ is the camera center. Wan2.2 temporally packs four neighboring frames to match the video VAE latent grid:

The original control adapter maps $Y_{\rm cam}$ to transformer-width features and adds them to the DiT patch tokens. This interface controls camera motion, but it says nothing about which animated surface lies on each ray.

DAR keeps the representation in Eq. (3), the temporal packing in Eq. (4), and the same residual injection point. The change is the control tensor:

where $B$ is a mesh-projected geometry buffer and $\mathcal{P}_{\rm vae}$ resamples it to the latent temporal grid. We replace the camera adapter by

The main configuration uses $N{=}9$. The adapter is the same lightweight SimpleAdapter family used by Wan2.2 camera control: pixel-unshuffle, stride-2 convolution, and two residual blocks, outputting transformer-width features. Camera and animated geometry are therefore fused before the DiT reasons over the video, rather than appended as late guidance.

For each visible pixel $u$ at time $t$, DAR projects three mesh-derived signals:

Here $q(s)\in[0,1]^{3}$ is a persistent tracking color or instance/part identity, $\bar{X}_{t}$ is the scene-normalized world position, and $\bar{n}_{t}$ is the normal mapped to image range. Pixels without a visible mesh hit are filled by the background convention used in the projection renderer and are masked consistently across all channels.

The three components serve different roles. Tracking gives a persistent identity cue for appearance transport. World position gives the current metric 3D state of the visible surface. Normal gives first-order local shape for shading and silhouette detail. We call this tensor a neural 4D G-buffer because it plays the role of a classical G-buffer (an image-format intermediate representation for rendering) but contains only the geometry needed by a reference-guided video diffusion renderer.

Depth is easy to render and widely supported by video models. With calibrated rays, an ideal geometric decoder can back-project depth into 3D; the issue is the inductive bias of the conditioning interface. A 4D renderer needs the condition image to expose both which persistent surface element is visible and where that element is in the target 3D state. These are different variables: appearance is attached to surface identity, while silhouette, occlusion, and camera-relative layout depend on current world state.

The reference image follows the Wan2.2 image-to-video inpainting interface. We place $I_{\rm ref}$ in the first frame of a video tensor, zero-fill the remaining frames, and provide a binary mask indicating which latent positions are known. This appearance condition is concatenated with the noisy latent in the original Wan2.2 path; DAR changes only the control adapter in Eq. (6).

Let $c=(I_{\rm ref},Y_{\rm DAR},\tau)$ collect all conditions. With flow-matching noise level $\sigma$, the model predicts the scheduler target $v_{\sigma}$:

For LoRA experiments, the pretrained Wan2.2 transformer is frozen except for rank-256 LoRA modules on attention/FFN projections and the new SimpleAdapter. The full-fine-tune variant trains all transformer parameters plus the same adapter. The 24-channel camera-only ablation is the same widened-interface code path with $N{=}0$.

Table 1 lists both raw assets and evaluated clips. This avoids conflating available rendering material with benchmark cases that have complete RGB, camera, geometry-buffer, inference, and metric verification.

Compared with existing 3D/4D resources, DAR-4D is organized around the paired tuple needed here: first-frame reference, target camera, animated mesh state, and dense geometry buffers. Appendix A.2 contrasts this interface with prior datasets.

The synthetic pipeline turns assets into plausible animated scenes through randomized but constrained assembly, complementing code-driven parametric generation of CAD and Blender scenes (Chen et al., 2026d). For the mixed-asset path, each scene samples multiple textured meshes, one or more rigged Z-OO animals, an HDRI environment, and a trajectory seed. Static meshes are scaled into a compact physical range, placed by AABB rejection sampling with a collision margin, and optionally given simple keyframed motion such as spin, tilt, bob, slide, sway, or self-orbit. Rigged animals are imported with their FBX animation, scaled to a target size, looped through NLA strips, and attached to drive empties so root motion can be composed with the skeletal walk cycle.

Motion planning is collision-aware. Initial placement avoids overlaps with existing static and animated objects. Walking paths are sampled as smooth piecewise trajectories in the ground plane, densified with Catmull-Rom interpolation, and rejected if the moving AABB intersects static objects or another moving path at the same frame. If no valid path is found, the asset is kept stationary rather than allowing interpenetration. This conservative rule keeps training conditions physically plausible.

Camera trajectories are generated from archetypes rather than fixed templates. The Blender path uses 16 trajectory families, including short/half/full orbits, orbit+dolly, top-down orbits, low arcs, pure dolly, lateral pan, rolling orbit, orbit+pan, and vertical arcs. The UE path uses a related 14-family generator with orbit, corridor, dolly, top-spin, and oblique motions. Each trajectory randomizes span, start yaw, pitch, roll, dolly, pan, and radius within conservative bounds. A fit-camera step computes animated scene bounds over sampled frames and clamps camera distance so the subject remains in view.

After rendering, every sample is converted into a unified DAR bundle. A sample directory contains video.mp4, the first-frame reference.png, camera intrinsics/extrinsics, renderer metadata, and per-frame projection files. Each projection stores five aligned signal families: tracking for dense object/part identity, world_pos for normalized world coordinates, normal for surface orientation, depth for a view-dependent baseline signal, and sparse_tracking for point-track style controls.

Quality control combines automatic filters with renderer-side constraints. The scripts reject or quarantine samples with near-uniform RGB, depth, normal, world-position, or tracking statistics, filter unsafe camera paths, verify scene-level world-position normalization, and inspect tracking/color buffers for consistency. These checks keep the reported benchmark conservative: the full corpus is designed to scale, but the paper reports only subsets with complete rendering, inference, and metric verification.

The main evaluation set is the 68-case 4d_vis benchmark: 17 base scenes and 4 camera trajectories per scene. All methods receive the same first-frame reference image, target camera trajectory, and mesh projection. Resolution is fixed to $480{\times}832$; sequence length follows the ground-truth clip and may be 49, 77, 81, 89, 93, 117, or 249 frames. We additionally report OOD-34, with 17 unseen scenes and 2 novel trajectories per scene, and a 10-case reference-variant subset that fixes geometry/camera while changing the reference image.

We compare four groups. Off-the-shelf Wan2.2 includes Wan2.2-Camera, conditioned on Plücker rays only, and Wan2.2-Depth, conditioned on ground-truth depth videos and the first-frame reference (Wan et al., 2025). DAR LoRA variants include the full condition with tracking, world position, and normal channels, together with 11 ablations. Single-channel rows isolate tracking, normal, world position, and depth; leave-one-out rows remove one channel family from the full design; depth-swap rows replace world position by depth in comparable channel sets. The complete configuration list is in Appendix A.3. Full fine-tune reports DAR at checkpoint 4000 without the LoRA bottleneck. External baselines include DAS and VideoFrom3D, executed on the same 68 cases (Gu et al., 2025; Kim et al., 2025).

DAS and VideoFrom3D have useful but mismatched interfaces: DAS is image/tracking-guided object-centric synthesis, while VideoFrom3D assumes sparse anchors and mostly static scene geometry. Their output length, resolution, and camera-control targets therefore do not exactly match our animated-mesh renderer protocol. We resize outputs and sample frames by normalized animation time before computing the same metrics, and mark these rows as diagnostic rather than part of the controlled ablation ranking.

We report PSNR, SSIM (Wang et al., 2004), and LPIPS (Zhang et al., 2018) as primary frame-aligned rendering metrics against the corresponding Blender ground-truth frame. We also report TempL1, the mean L1 distance between adjacent-frame differences, and RefL1, the mean per-frame L1 distance to the first-frame reference. RefL1 is auxiliary: a low value can indicate appearance retention, but it can also indicate that a method stays too close to the first view rather than respecting novel camera motion. We leave tracking IoU, ATE/RTE, FVD, and VBench-style scores to future work because they require reverse-estimated masks, cameras, or perceptual judgments that would introduce additional estimator error into the current frame-aligned protocol.

All LoRA settings start from Wan2.2-Fun-5B-Control-Camera. The backbone is frozen except for rank-256 LoRA modules on attention QKV/O and FFN projections, while the new SimpleAdapter is trained. We use AdamW, learning rate $1{\times}10^{-4}$, constant-with-warmup scheduling with 200 warmup steps, batch size 1, gradient accumulation 1, 10,000 training steps, checkpoints every 2,000 steps, bf16 mixed precision, and gradient checkpointing. Inference uses guidance scale 3.0, 50 flow-matching denoising steps, and seed 42. Each LoRA run uses either $3{\times}$ A800-80GB or $2{\times}$ H100-80GB and takes roughly 36 hours.

For the full fine-tune, we train selected condition sets without LoRA on GPUs 1–7 ($7{\times}$ A800-80GB) using FSDP Full-Shard. The backbone and SimpleAdapter are trained together, for roughly 5.4B trainable parameters. Hyperparameters match the LoRA recipe except for a lower learning rate, $2{\times}10^{-5}$. Sec. 6.3 reports DAR checkpoint 4000 on all 68 4d_vis cases; Appendix A.4 reports the matched checkpoint-2000 full-fine-tune ablations.

Table 2 gives the 68-case comparison. All LoRA rows are reported at checkpoint 10,000. Rows marked with $\dagger$ use normalized-time alignment because the external system emits a non-matching video length or resolution; we include them as diagnostic reference points, but exclude them from the controlled rank ordering.

The post-reference figure pages show representative outputs. Fig. 4 shows the final-frame comparison against Wan2.2-Camera, Wan2.2-Depth, and Blender GT under a $\pm 60^{\circ}$ orbit, where the target view is farthest from the reference: Wan2.2-Camera hallucinates the foreground and Wan2.2-Depth couples camera and object motion, whereas DAR matches both the target camera and the animated mesh state. Fig. 5 compares Wan2.2-Camera, Wan2.2-Depth, DAS, VideoFrom3D, and DAR at matched normalized animation times. Camera-only control follows viewpoint but hallucinates the foreground; depth preserves coarse layout but entangles camera and object motion; DAS preserves some local identity but is object-centric; VideoFrom3D can synthesize plausible novel views but is tied to a mostly static-scene assumption. DAR better follows both the target camera and the animated mesh state. Fig. 3 shows that the same interface handles humans, animals, vehicles, mechanical objects, and indoor scenes under large horizontal orbits.

Appendix A.4 adds four stress tests beyond Table 2. First, trajectory-stratified results show that DAR is best on all four camera-motion groups, including the largest $\pm 60^{\circ}$ orbit. Second, the matched depth-swap comparison favors world position at every LoRA checkpoint from 2k to 10k. Third, replacing persistent tracking by a binary foreground mask at checkpoint 2000 drops PSNR by 1.60 dB under LoRA and 1.27 dB under full fine-tuning. Fourth, OOD-34 remains a stress test: Wan2.2-Depth reaches 22.25 PSNR, tracking+world-position conditioning reaches 20.99, and DAR reaches 19.04, indicating that the full 9-channel LoRA model still needs more capacity or data for open-category generalization.

The reference-variant subset tests appearance control by fixing one animated car mesh and camera while changing only the reference image, as shown in Fig. 1. DAR changes paint style, color blocks, and vehicle appearance while keeping the camera, pose, and silhouette aligned, indicating partial decoupling between geometry and appearance. To estimate the ceiling of DAR without the LoRA bottleneck, we also fully fine-tune the model. Checkpoint 4000 reaches PSNR 25.36, SSIM 0.917, LPIPS 0.130, and TempL1 4.72 on all 68 cases: +2.14 dB above LoRA DAR ck10000 and +3.68 dB above off-the-shelf Wan2.2-Depth.