2,000 Hours of Smartphone-Collected Egocentric Manipulation Data
with a Complete Data-to-Model Toolchain
English | 简体中文
Tip
New to Open-AoE? Start with the data specification, render one segment with AoE-Visualization, then choose a model recipe from AoE-Training-Ready. Use DTCoder to start your experiment.
Open-AoE is a large-scale, real-world egocentric manipulation dataset collected entirely with consumer smartphones. It provides approximately 2,000 hours of first-person video with synchronized hand motion, camera motion, and bilingual atomic-action annotations. The repository connects those signals to visualization, human-to-robot retargeting, robot replay, and model-specific training recipes.
| Resource | Access | What you get |
|---|---|---|
| Technical report | Read on arXiv | Dataset design, processing, analysis, toolchain, and experiments |
| Open-AoE-2000H | Hugging Face · ModelScope | Dataset files and distribution information |
| Data specification | Field-level documentation | Directory layout, schemas, coordinate systems, and validation notes |
| Capture app | Search for “具身智能数采助手” in major app stores | Smartphone data collection client |
The technical report is available on arXiv.
| I want to… | Start here | Result |
|---|---|---|
| Understand what is in one segment | Dataset specification | Learn the video, calibration, MANO, camera-trajectory, and annotation fields |
| See the data before writing code | AoE-Visualization | Render an end-to-end review video with hands, trajectories, actions, and a 3D view |
| Train a VLA policy | LeRobot recipes · GR00T N1.7 · H-RDT · VITRA | Convert Open-AoE signals to model-specific state/action semantics and launch training |
| Train a world or video-action model | Training recipe index | Use DreamZero, LingBot-VA, Ctrl-World, iVideoGPT, GenieRedux, LAOM, AdaWorld, or DreamDojo integrations |
| Reconstruct scenes or retarget human motion | AoE-Reconstruct-Retarget | Reconstruct interaction assets and produce robot trajectories, simulation renders, or robotized video |
| Add a model or robot integration | Contributing guide | Follow the repository structure and dependency rules |
git clone https://github.com/ant-research/Open-AoE.git
cd Open-AoEChoose either Hugging Face or ModelScope, then locate one extracted segment directory. Its expected layout is documented in the data specification.
Some visualization and retargeting workflows require MANO model files. Register and download MANO_RIGHT.pkl and MANO_LEFT.pkl from the MANO website, then install them into the repository's shared asset directory:
bash assets/mano/download_mano.sh \
~/Downloads/MANO_RIGHT.pkl \
~/Downloads/MANO_LEFT.pkl
cd aoe-visualization
pip install -r requirements.txt
python visualize.py --sample /path/to/open_aoe_segmentThe output is output/<segment-name>/AoE_output_vis.mp4, a synchronized review of the ego video, reconstructed hands, wrist trajectories, atomic actions, world-frame motion, and timeline. See the visualization guide for EGL/OpenGL requirements and batch rendering.
Each segment is a synchronized multimodal record rather than a standalone video:
| Signal | Main artifact | Typical use |
|---|---|---|
| Raw and undistorted RGB | raw_video.mp4, raw_video_undistorted.mp4 |
Visual observation, video modeling, and overlays |
| Camera metadata | video_info.json, undistorted_video_info.json |
Intrinsics, distortion, device information, resolution, and frame rate |
| Camera motion | camera_traj.npz and transforms in hands.npz |
Metric-scale 6-DoF trajectories and world/camera transforms |
| Hand reconstruction | hands.npz |
Per-frame bilateral MANO pose, shape, root transform, and validity |
| Atomic actions | ego_action_annotation.json |
Temporally aligned actions with hand, verb, object, and bilingual description |
The toolchain then maps those synchronized signals into task-specific representations:
| Stage | Component | Output |
|---|---|---|
| Inspect | AoE-Visualization | One review video per segment for visual and temporal quality checks |
| Reconstruct / retarget | AoE-Reconstruct-Retarget | Reconstructed assets, robot trajectories, simulation validation, and robotized video |
| Convert / train | AoE-Training-Ready | Model-specific datasets, actions, patches, launchers, and training recipes |
Important
Training conversion is an action-semantics adaptation, not just file-format conversion. Start with the shared action specification, then follow the README for your target model.
| Target | Included recipes | Recommended entry |
|---|---|---|
| VLA policies | ACT, Diffusion Policy, π0.5, SmolVLA, GR00T N1.7, H-RDT, VITRA | Training-Ready index |
| World / video-action models | DreamZero, LingBot-VA, Ctrl-World, iVideoGPT | Training-Ready index |
| Latent-action / world models | GenieRedux, LAOM, AdaWorld, DreamDojo | Training-Ready index |
Each recipe is self-contained and documents its upstream repository and verified commit, data conversion, environment variables, training command, outputs, and any required patch. Upstream projects and checkpoints are not vendored into this repository.
| Subproject | Coverage | Main capabilities |
|---|---|---|
| Phantom | Unitree G1 + Dex3 / Inspire | Arm IK, dexterous-hand retargeting, MuJoCo visualization, and robot overlay |
| Retarget Galbot | Galbot / Galaxea bimanual platforms | Palm-to-TCP IK, gripper mapping, egoview synthesis, and LeRobot/Rerun export |
| AoE Retarget Lab | EgoInfinity/G1, Do-as-I-Do/Sharpa, SPIDER/XHand | External-method adapters, 6-DoF reconstruction routes, and a 12-cell comparison matrix |
Third-party repositories, model weights, robot assets, and generated videos are not included. Follow each subproject's setup guide to obtain its external dependencies.
- On-device capture control checks hand visibility, wearing conditions, lighting, motion quality, and device health before upload.
- Offline quality control and scene labeling filters invalid or sensitive content, standardizes frame rate, slices videos, and assigns scene/task metadata.
- Reconstruction and annotation estimates camera trajectories, reconstructs MANO hands, and produces atomic-action segments.
- Quality inspection and delivery applies completeness, correctness, and temporal-consistency gates followed by human review.
Open-AoE/
├── Open-AoE-tech-report.pdf # Current technical report
├── open-aoe-2000h/ # Dataset format and field-level documentation
├── aoe-visualization/ # Synchronized data review and rendering
├── aoe-reconstruct-retarget/ # Reconstruction, retargeting, replay, and overlays
├── aoe-training-ready/ # Model adapters, converters, launchers, and patches
├── assets/mano/ # Shared MANO setup helper; model files are not tracked
├── docs/ # Overview and pipeline figures
├── CONTRIBUTING.md
├── LEGAL.md
└── LICENSE
- Smartphone-first collection: consumer devices make real-world egocentric capture accessible and scalable.
- Manipulation-aligned signals: calibrated RGB is synchronized with camera motion, MANO hand reconstruction, validity masks, and atomic actions.
- Complete data-to-model path: the repository covers inspection, retargeting, representation conversion, and model-specific training integration.
- Modular and extensible: components can be used independently, and new robot embodiments or model recipes can be added as self-contained integrations.
The technical report is available on arXiv.
- Data processing pipeline applications: kaile.zk@antgroup.com
- Industry collaboration: kaile.zk@antgroup.com
Contributions of robot embodiments, reconstruction or retargeting backends, visualization features, data converters, and training recipes are welcome. Read CONTRIBUTING.md before opening a pull request.
Original source code in this repository is released under the Apache License 2.0. Dataset distribution terms, model weights, robot assets, MANO files, and third-party components may use different licenses. Read LEGAL.md and the relevant subproject README before redistribution or commercial use.
Open-AoE is built by the AoE community. We gratefully acknowledge the following contributors from industry, universities, and research institutions.
- National University of Singapore: Qingze Guan
- University of Chinese Academy of Sciences: Zhengxing Wu
- Ant Digital Technology, Ant Group: Zishuo Li, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan
- Zhejiang University: Jiadong Hong
- The University of Hong Kong: Zhenchao Jin, Yushi Feng
- The Hong Kong University of Science and Technology (Guangzhou): Taowen Wang
- Beijing Academy of Artificial Intelligence: You Liu, Yibo Wang
- University of Chinese Academy of Sciences: Yifan Yang
- The Hong Kong University of Science and Technology: Hao Cheng
- Ant Digital Technology, Ant Group: Bowen Yang, Changtao Miao, Zhaowen Zhou, Man Luo
We also thank all data contributors, maintainers, and upstream open-source and research communities whose work made this release possible.