ReWorld: An Interactive World Model with Long-Horizon Memory 🌍

_{▶ Click the preview to watch the demo reel with sound (HD file: assets/demo/reworld_demo.mp4).}

ReWorld is an interactive world model: you drive the camera with keyboard-and-mouse actions, and it streams the world back to you chunk by chunk. Its window-split training scheme decouples control from memory, so precise action following and long-horizon consistency are learned without competing against each other. At inference, a bounded KV cache paired with a pose-indexed landmark bank keeps GPU memory constant regardless of rollout length, while still retrieving the right past views when the camera revisits a place. Trained on metric-aligned multi-source data and distilled to 4 denoising steps, ReWorld streams 704×1280 video in real time.

• 2026-08 — Paper released on arXiv: ReWorld: An Interactive World Model with Long-Horizon Memory.
• 2026-08 — Inference code released. Pretrained checkpoints are on the way — see Checkpoints.

Tested with Python 3.10 and PyTorch ≥ 2.4 on CUDA GPUs.

git clone https://github.com/zhifeichen097/ReWorld.git
cd ReWorld

conda create -n reworld python=3.10 -y
conda activate reworld

pip install -r requirements.txt
pip install flash-attn --no-build-isolation   # required by the attention kernels
pip install peft                              # required for the 4-step DMD LoRA

Notes:

• flash-attn (v2 or v3) is required — the cross-attention path calls it directly.
• peft is only needed when loading the few-step LoRA (the default config does).

ReWorld weights are not released yet. The table below lists what will be published; until then, the inference commands cannot be run end-to-end. The Wan2.2 base model is already public and can be downloaded today.

Expected layout (paths are set in configs/plucker720p_dmd_infer.yaml; the base-model directory can also be pointed to via the WAN_MODEL_DIR environment variable):

checkpoints/
├── Wan2.2-TI2V-5B/            # official Wan2.2 release (T5, tokenizer, VAE)
├── reworld_generator_ema.pt   # coming soon
└── reworld_dmd_lora.pt        # coming soon

Two entry points, each with a plain and a _v2 variant. Use the _v2 scripts — they swap in the improved landmark bank (bounded memory, top-k pose retrieval) and the 720p decode memory fix, with an identical CLI.

Rolls out every prompt@keysequence line in assets/mc_eval_random_keys_96latent_more7.txt (set in the config), conditioned on your start image, with the pose-indexed landmark bank capping the KV cache at 12 chunks (1 sink + 5 recent + 6 landmarks retrieved from a 30-entry bank):

python inference_i2v_v2.py \
    --config_path configs/plucker720p_dmd_infer.yaml \
    --mode dataset \
    --init_image path/to/start_image.png \
    --num_inference_steps 4 \
    --kv_policy v15b \
    --kv_budget_chunks 12 \
    --kv_n_sink 1 \
    --kv_recent_w 5 \
    --output_folder outputs/reworld_i2v

On GPUs where 720p VRAM is tight, prefix with BANKV2_HOT_GPU=0 to keep the bank in pinned CPU memory.

Type one mouse key (i/k/j/l/u — look up/down/left/right/none) and one keyboard key (w/a/s/d — translate, or q — stay) per 4-latent-frame chunk:

python inference_action_v2.py \
    --config_path configs/plucker720p_dmd_infer.yaml \
    --mode interactive \
    --prompt "A cinematic Minecraft village at sunset" \
    --num_inference_steps 4 \
    --output_folder outputs/interactive

torchrun --nproc_per_node=4 inference_action_v2.py \
    --config_path configs/plucker720p_dmd_infer.yaml \
    --mode dataset \
    --num_inference_steps 4 \
    --output_folder outputs/eval_action

The full flag reference — trajectory key-string format, all memory policies and their knobs, environment variables, ablation switches — is in docs/INFERENCE.md.

Training splits each video window so that action-conditioned generation and memory-conditioned generation are supervised separately, which decouples control fidelity from long-horizon recall. At inference, a bounded KV cache holds sink and recent chunks at full resolution, while a landmark bank indexed by camera pose stores distinct past viewpoints and retrieves the top-k nearest ones for attention — memory stays constant while the world stays consistent.

ReWorld is built on the Wan2.2 backbone (DiT, VAE, and text encoder) and adapts the causal-distillation codebase of Self-Forcing. We thank the authors of both projects for open-sourcing their work.

If you find ReWorld useful, please cite:

@article{chen2026reworld,
  title   = {ReWorld: An Interactive World Model with Long-Horizon Memory},
  author  = {Chen, Zhifei and Wang, Luozhou and Shen, Guibao and Yan, Dongyu and
             Yang, Shuai and Xu, Tianshuo and Du, Yihua and Wang, Wei and
             Gui, Tianyi and Huang, Lianghua and Chen, Yingcong},
  journal = {arXiv preprint arXiv:2608.23565},
  year    = {2026}
}

This project is released under the CC BY-NC-SA 4.0 license, for research and non-commercial use only. The Wan2.2 base model is subject to its own license terms.