2,000 Hours of Smartphone-Collected Egocentric Manipulation Data
with a Complete Data-to-Model Toolchain

English | 简体中文

Open-AoE is a large-scale, real-world egocentric manipulation dataset collected entirely with consumer smartphones. It provides approximately 2,000 hours of first-person video with synchronized hand motion, camera motion, and bilingual atomic-action annotations. The repository connects those signals to visualization, human-to-robot retargeting, robot replay, and model-specific training recipes.

The technical report is available on arXiv.

git clone https://github.com/ant-research/Open-AoE.git
cd Open-AoE

Choose either Hugging Face or ModelScope, then locate one extracted segment directory. Its expected layout is documented in the data specification.

Some visualization and retargeting workflows require MANO model files. Register and download MANO_RIGHT.pkl and MANO_LEFT.pkl from the MANO website, then install them into the repository's shared asset directory:

bash assets/mano/download_mano.sh \
  ~/Downloads/MANO_RIGHT.pkl \
  ~/Downloads/MANO_LEFT.pkl

cd aoe-visualization
pip install -r requirements.txt
python visualize.py --sample /path/to/open_aoe_segment

The output is output/<segment-name>/AoE_output_vis.mp4, a synchronized review of the ego video, reconstructed hands, wrist trajectories, atomic actions, world-frame motion, and timeline. See the visualization guide for EGL/OpenGL requirements and batch rendering.

Each segment is a synchronized multimodal record rather than a standalone video:

The toolchain then maps those synchronized signals into task-specific representations:

Each recipe is self-contained and documents its upstream repository and verified commit, data conversion, environment variables, training command, outputs, and any required patch. Upstream projects and checkpoints are not vendored into this repository.

Third-party repositories, model weights, robot assets, and generated videos are not included. Follow each subproject's setup guide to obtain its external dependencies.

1. On-device capture control checks hand visibility, wearing conditions, lighting, motion quality, and device health before upload.
2. Offline quality control and scene labeling filters invalid or sensitive content, standardizes frame rate, slices videos, and assigns scene/task metadata.
3. Reconstruction and annotation estimates camera trajectories, reconstructs MANO hands, and produces atomic-action segments.
4. Quality inspection and delivery applies completeness, correctness, and temporal-consistency gates followed by human review.

Open-AoE/
├── Open-AoE-tech-report.pdf  # Current technical report
├── open-aoe-2000h/           # Dataset format and field-level documentation
├── aoe-visualization/        # Synchronized data review and rendering
├── aoe-reconstruct-retarget/ # Reconstruction, retargeting, replay, and overlays
├── aoe-training-ready/       # Model adapters, converters, launchers, and patches
├── assets/mano/              # Shared MANO setup helper; model files are not tracked
├── docs/                     # Overview and pipeline figures
├── CONTRIBUTING.md
├── LEGAL.md
└── LICENSE

• Smartphone-first collection: consumer devices make real-world egocentric capture accessible and scalable.
• Manipulation-aligned signals: calibrated RGB is synchronized with camera motion, MANO hand reconstruction, validity masks, and atomic actions.
• Complete data-to-model path: the repository covers inspection, retargeting, representation conversion, and model-specific training integration.
• Modular and extensible: components can be used independently, and new robot embodiments or model recipes can be added as self-contained integrations.

The technical report is available on arXiv.

• Data processing pipeline applications: kaile.zk@antgroup.com
• Industry collaboration: kaile.zk@antgroup.com

Contributions of robot embodiments, reconstruction or retargeting backends, visualization features, data converters, and training recipes are welcome. Read CONTRIBUTING.md before opening a pull request.

Original source code in this repository is released under the Apache License 2.0. Dataset distribution terms, model weights, robot assets, MANO files, and third-party components may use different licenses. Read LEGAL.md and the relevant subproject README before redistribution or commercial use.

Open-AoE is built by the AoE community. We gratefully acknowledge the following contributors from industry, universities, and research institutions.

• National University of Singapore: Qingze Guan
• University of Chinese Academy of Sciences: Zhengxing Wu
• Ant Digital Technology, Ant Group: Zishuo Li, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan

• Zhejiang University: Jiadong Hong
• The University of Hong Kong: Zhenchao Jin, Yushi Feng
• The Hong Kong University of Science and Technology (Guangzhou): Taowen Wang
• Beijing Academy of Artificial Intelligence: You Liu, Yibo Wang
• University of Chinese Academy of Sciences: Yifan Yang
• The Hong Kong University of Science and Technology: Hao Cheng
• Ant Digital Technology, Ant Group: Bowen Yang, Changtao Miao, Zhaowen Zhou, Man Luo

We also thank all data contributors, maintainers, and upstream open-source and research communities whose work made this release possible.