VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End? [Project Page]

VWE-Bench Dataset  |  VibeWorlder Models  |  VibeWorlding Paper

VibeWorlding is a unified open-source framework for benchmarking and training vibe worlding agents: the multimodal agents that autonomously infer user intent, plan a scene layout, invoke 3D tools, and reflect on multimodal feedback (the 3D map plus rendered images) over multi-turn agent–environment interaction.

• 2026-08-16 — We release VWE-Bench, VibeWorlder-Embedding-4B, VibeWorlder-8B, VibeWorlder-30B-A3B, and our technical report.

The vibeworlding agent handles two task families end-to-end: 3D world construction (build a 3D world from scratch) and 3D world refinement (refine an existing 3D world). The illustrative examples are here:

3D World Construction — build a 3D world from scratch:

Example query: Create a simple farm with a barn, crop fields, fences, and a few trees.

3D World Refinement — refine an existing 3D world:

Example query: Remove the car in the middle of the road and delete the two green buildings.

The framework has two halves that share the same Blender sandbox. VWE-Bench (left) is the evaluation suite — 2,616 curated 3D assets, 323 human-annotated seed worlds, and 6,828 reverse-synthesized queries spanning world construction and refinement — scored by a rubric-based verifier. VibeWorlding-Gym (right) is the training framework: the same sandbox is exposed to the agent as MCP-style tools, and the same verifier is used as the reward service for joint multimodal RL post-training.

How the agent works. Each turn the agent observes the current 3D map plus 5 rendered views, then calls tools — retrieve_assets, add, delete, rotation_and_translation. The scene is re-rendered and fed back, so the agent can see what it built and fix it.

The role of every box in the figure maps to a concrete component in this repo:

VibeWorlding-Gym/
├── main.py                  # agent sampling (generate / refine)
├── eval.py                  # evaluation entry point (3 verifier routes)
├── render_raw_data_images.py# batch-render 5 views for a data directory
├── start_verl_server.py     # serve a local model with vLLM (eval/sampling)
├── start_verl_server_CLI.py # serve a local model with vLLM (for the CLI)
├── assets_retrieval/        # ▸ asset retrieval service   (see its README)
│                            #   models/ ships VibeWorlder-Embedding-4B
├── render_in_blender/       # ▸ PCG rendering service     (see its README)
│                            #   assets/models/clone/ ships the GLB assets
├── utils/                   # LLM clients, agent tools, prompts,
│                            #   map_parser.py (3D map → PCG actors),
│                            #   broker.py (concurrent verifier RPC), scene utils
├── verifier/                # rubric + rule-based verifiers
├── baseline/                # SceneWeaver / SAGE / SceneAssistant
├── CLI_Demo/                # VibeWorld interactive CLI   (see its README)
├── verl/                    # SFT + RL training
├── data/
│   ├── sft/                 # 5,567 SFT cases (1,129 construction + 4,438 refinement)
│   ├── test/                # 254 evaluation cases
│   ├── rl/                  # RL parquet (train/test)
│   └── seed_3dworld/        # seed 3D worlds (fill from VWE-Bench)
└── log/                     # sampling / eval outputs

VWE-Bench is what the agent is benchmarked and trained on. It is built from three ingredients, all shipped in this repo. You can preview the assets in via here

Assets & seed worlds. A curated library of 2,616 GLB assets (furniture, vegetation, buildings, vehicles, food, … — each with a 5-digit type_id, display name, scale class, and per-asset orientation correction) and 323 human-annotated seed 3D worlds (8–258 placed assets per world, median 28, ranging from riverside hamlets to dense villa towns). The agent works with the exact same geometry as the benchmark.

Queries. 6,828 reverse-synthesized multimodal queries split into two task families. Only asset-level edit (precise) has a ground-truth map and is verified (rule-based); the rest are unverified and scored by an MLLM judge against our rubrics.

Every query.json carries query_category + query_type (the labels above), plus query_tag for internal bookkeeping and verifier_type (verified / unverified) which selects the scoring route (see §4).

Data splits in this repo:

The three splits use completely disjoint seed worlds (264 SFT / 47 RL / 12 Test seeds, summing to 323), so leaderboard numbers measure generalization rather than seed memorization.

The agent cannot do anything until the sandbox is up: (a) the two services — PCG rendering and asset retrieval — and (b) the training stack under verl/. Set them up in this order.

Runtime: Python 3.12 with torch 2.10.0+cu129, vllm 0.17.0, transformers 4.57.6. The services and the training stack can live on separate machines. API keys are read from the environment — nothing is hardcoded.

Follow render_in_blender/README.md — install Blender 4.2.x and place the GLB assets (assets/models/clone/, download from VWE-Bench if not already present). Rendering is GPU-bound, so run several workers per GPU:

# PCG rendering  ->  :8080
# WORKERS_PER_GPU=8 starts 8 render workers per GPU behind a sticky proxy on :8080.
# Use WORKERS=1 for a quick single-worker install check.
cd render_in_blender && BLENDER_EXE=/opt/blender-4.2.0-linux-x64/blender \
  WORKERS_PER_GPU=8 PORT=8080 bash deploy.sh

Verify — render the bundled example scene into 5 views:

python render_raw_data_images.py \
  --raw_data_dir ./render_in_blender/assets/cmds/ \
--server http://localhost:8080 \
  --quality "低质量 (快速预览)"

Follow assets_retrieval/README.md — place the VibeWorlder-Embedding-4B model (from the model collection), then launch:

# asset retrieval  ->  :8081
cd assets_retrieval && PORT=8081 bash deploy.sh

Verify — a text query should return ranked assets with 5-digit type_id:

curl -s -X POST "http://localhost:8081/recommend/single_slot" \
  -H "Content-Type: application/json" \
  -d '{"entity_name": "探险木屋", "top_k": 3}' | python3 -m json.tool

Point the agent / client at both services:

export VIBEWORLD_RENDER_SERVER=http://localhost:8080
export VIBEWORLD_RETRIEVE_SERVER=http://localhost:8081

# Please install this environment separately from the PCG/Retrieval environments.

pip install openai google-genai gradio-client httpx requests pillow numpy

# training stack (SFT + RL on verl)
pip install -r verl/requirements.txt

export GEMINI_API_KEY=your_gemini_api_key
export OPENAI_API_KEY=your_openai_api_key
export DASHSCOPE_API_KEY=your_dashscope_api_key   # Bailian / DashScope

The same loop is used for both leaderboard evaluation and for SFT/RL trajectory generation: an agent reads a query, calls tools over multi-turn interaction, and the resulting scene is scored by the verifier. To reproduce the VWE-Bench leaderboard below, run main.py then eval.py over data/test/ (254 cases).

The two panels show the same models on the two query families introduced in §2. Verified (left) is rule-based and tests whether the model can hit a known target map. Unverified (right) is judged by an MLLM against intent rubrics and is where training pays off most — note how the VibeWorlder models (the last two bars in each panel, ours) lead on the unverified split after joint multimodal RL post-training. Use this figure to pick a model:

• Strongest public baseline: GPT-5.5 leads on Verified (rule-following), Gemini-3.5-flash leads on Unverified (intent following).
• Our 8B / 30B models (VibeWorlder-8B and VibeWorlder-30B-A3B) are competitive with the strongest closed models on Verified and beat them on Unverified after post-training.
• Pick your serving cost — the small open VibeWorlder-8B is a good default for main.py; the larger 30B is what the verl/ recipes train.

With both services configured, roll out an agent over a data directory. The task type (generate vs refine) is auto-detected per case from query.json.

python main.py \
  --base_data_dir data/test \
  --log_dir log/eval_test \
  --model_type gemini --model_name gemini-3.1-pro \
  --server http://localhost:8080 \
  --retrieve_server http://localhost:8081 \
  --quality "低质量 (快速预览)" \
  --max_turns 8

--model_type accepts gemini, openai, qwen3, bailian, offline-llm. Use offline-llm for a locally served model:

python start_verl_server.py --model_path ./models/VibeWorlder-30B-A3B --tp_size 4 --port 8000
export VERL_SERVER_URL=http://localhost:8000
python main.py --base_data_dir data/test --log_dir log/eval_ours \
  --model_type offline-llm --model_name vibeworlder-30B-A3B

Handy flags: --max_cases N (smoke test), --cases 001,002, --task_setting, --debug.

Each case writes final_map.json, sft_trajectory.json, and final_image/ into --log_dir.

python eval.py \
  --result_dir log/eval_test \
  --model_type gemini --model_name gemini-3.5-flash

Cases are dispatched to one of three scoring routes automatically. These route names are an implementation detail of the verifier. The mapping to the §2 query categories is:

SFT is rejection-sampled: we roll out trajectories with a strong teacher, keep only the ones that pass the verifier, and train on those. To skip the sampling and packing yourself, we release a ready-to-train parquet example at

→ huggingface.co/datasets/usail-hkust/VWE-Bench — data/sft_parquet_example

It has the same three columns the trainer expects — messages (the turn-by-turn agent transcript with the RL-aligned system prompt from utils/prompt.py::get_system_prompt), images (JPEG bytes downscaled for Qwen3-VL), and tools (the 4 tool schemas) — with train.parquet / val.parquet already split. Point DATA_DIR at it and go:

huggingface-cli download usail-hkust/VWE-Bench \
  --repo-type dataset \
  --include "data/sft_parquet_example/*" \
  --local-dir ./data/sft_packed_download
# resulting parquets live under ./data/sft_packed_download/data/sft_parquet_example/

cd verl

# single node
DATA_DIR=../data/sft_packed_download/data/sft_parquet_example bash run_map_gen_sft.sh

# multi node (run on every node, varying NODE_RANK)
NODE_RANK=0 MASTER_ADDR=<master-node-ip> DATA_DIR=../data/sft_packed_download/data/sft_parquet_example bash run_map_gen_sft_multinode.sh
NODE_RANK=1 MASTER_ADDR=<master-node-ip> DATA_DIR=../data/sft_packed_download/data/sft_parquet_example bash run_map_gen_sft_multinode.sh
NODE_RANK=2 MASTER_ADDR=<master-node-ip> DATA_DIR=../data/sft_packed_download/data/sft_parquet_example bash run_map_gen_sft_multinode.sh

Everything is overridable by environment variable, so you should not need to edit the scripts. MODEL_ID defaults to the base model under MODEL_HOME (Qwen3-VL-8B-Thinking single node, Qwen3-VL-30B-A3B-Thinking multi node):

VIBEWORLD_ROOT=/path/to/VibeWorlding-Gym \
MODEL_HOME=/path/to/models \
MODEL_ID=/path/to/Qwen3-VL-8B-Thinking \
DATA_DIR=/path/to/sft_parquet_example \
CKPT_HOME=/path/to/output \
bash run_map_gen_sft.sh

The resulting checkpoint then warm-starts RL — see the next section.

Joint multimodal RL post-training on top of an SFT checkpoint. The figure shows the key empirical finding: cold-starting RL from the base model (solid) learns slowly and flattens early, while initializing from the SFT checkpoint (dashed) climbs steadily and pulls ahead on every split — most dramatically on the verification set, where the reward more than doubles. We recommended override is your SFT output:

HF_MODEL_PATH=./models/ckpt/map_gen_sft/<exp>/global_step_N/actor/huggingface \
bash run_map_gen_grpo.sh

Multimodal agentic RL where the reward comes from the same verifier used for evaluation, and rollouts call the retrieval + rendering services. Start both services first. RL throughput is gated by the renderer, so run it with several workers per GPU — WORKERS_PER_GPU=8 bash deploy.sh (64 workers on an 8-GPU node); see render_in_blender/README.md.

verified queries are scored by rule against gt_map (no LLM at all). unverified queries need an MLLM judge, and there are two transports for it:

cd verl
export RETRIEVE_SERVER_URL=http://localhost:8081
export PCG_GRADIO_SERVER=http://localhost:8080
export VERIFY_MODEL_TYPE=gemini            # gemini | openai | qwen3 | bailian | offline-llm
export VERIFY_MODEL_NAME=gemini-3.5-flash  # recommended judge for RL
export GEMINI_API_KEY=your_gemini_api_key
export WANDB_API_KEY=your_wandb_api_key    # optional

bash run_map_gen_grpo.sh                                            # single node
NODE_RANK=0 MASTER_ADDR=<master-node-ip> bash run_map_gen_grpo_multinode.sh

RL issues a judge call for every unverified rollout, so latency and cost dominate throughput, and the rubric is coarse enough that flash tracks pro closely.

RL starts from the base model by default (Qwen3-VL-8B-Thinking single node, Qwen3-VL-30B-A3B-Thinking multi node, resolved under MODEL_HOME, default ./models). To warm-start from an SFT checkpoint instead:

HF_MODEL_PATH=./models/ckpt/map_gen_sft/<exp>/global_step_N/actor/huggingface \
bash run_map_gen_grpo.sh

A batch of rollouts finishing together would otherwise each block on its own serial API call. The broker accepts all those requests at once and runs them through a thread pool, so judging happens concurrently. Requests within a single session stay serialized, preserving conversation order.

On the networked machine (holds the API key, shares the same mount):

export GEMINI_API_KEY=your_gemini_api_key
cd utils
BROKER_WORKERS=16 ./start_broker.sh     # background, auto-restarts on crash
./start_broker.sh status                # also: log | stop
./start_broker.sh --echo                # echo mode: exercise the plumbing, no LLM

On the training node:

# single node
cd verl
VIBEWORLD_LLM_TRANSPORT=filerpc bash run_map_gen_grpo.sh

# multi node (run on every node, varying NODE_RANK)
NODE_RANK=0 MASTER_ADDR=<master-node-ip> VIBEWORLD_LLM_TRANSPORT=filerpc bash run_map_gen_grpo.sh
NODE_RANK=1 MASTER_ADDR=<master-node-ip> VIBEWORLD_LLM_TRANSPORT=filerpc bash run_map_gen_grpo.sh
NODE_RANK=2 MASTER_ADDR=<master-node-ip> VIBEWORLD_LLM_TRANSPORT=filerpc bash run_map_gen_grpo.sh

An interactive terminal agent — describe a world and watch it get built, with a live browser 3D viewer. See CLI_Demo/README.md.

pip install -e CLI_Demo
vibeworld

Download our models from the VibeWorlder collection and serve them locally:

huggingface-cli download usail-hkust/VibeWorlder-30B-A3B \
  --local-dir ./models/VibeWorlder-30B-A3B

python start_verl_server_CLI.py \
  --model_path ./models/VibeWorlder-30B-A3B --tp_size 4 --port 8000

export VIBEWORLD_LOCAL_VLLM_URL=http://localhost:8000/v1
vibeworld --model vibeworlder

All providers stream token-by-token. Switch at runtime with /model: our local models (vibeworlder), Gemini official (gemini-flash, gemini-pro), OpenAI official (gpt5), and Bailian / DashScope (qwen, k3).

If you use the VibeWorlding-Gym framework, VWE-Bench dataset, or substantially derived materials from this project in your research, please cite the following paper:

@article{vibeworlding2026,
  title   = {VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?},
  author  = {Ning, Yansong and Ye, Jingwen and Wu, Zhongkai and Sun, Yang and
             Zhu, Yiqin and Li, Xingyi and Zhang, Weidong and Liu, Hao},
  journal = {arXiv preprint},
  year    = {2026}
}

The VWE-Bench dataset is released under the Apache License 2.0 as indicated on its Hugging Face dataset card.

This repository may include or depend on third-party software, models, datasets, 3D assets, or other materials that are subject to their own licenses and terms. Such third-party materials are not relicensed by this NOTICE or by the Apache License applicable to the original VibeWorlding code. Users are responsible for complying with the applicable licenses and terms for such third-party materials.