Paper (arXiv:2606.19333) · Official code (malik-group/do-as-i-do) · Project page

An independent engineering study of the official Do as I Do pipeline (dexterous-manipulation data from everyday human videos): we migrated the upstream conda-based environments (python 3.10–3.12, torch 2.7–2.12) onto one standard modern stack — Python 3.11 + torch 2.7.0+cu128, uv venvs only — and validated every stage end to end on the shipped whisking demo, including the VRAM-hungry SAM3D stages on a remote 4×A800 server. Scope is simulation-only: deployment/ (real UR3e + Sharpa Wave robot) is included for reference but was not exercised.

This is not a fork for development — it is a reproducibility artifact: the official code tree (with 3 small env-migration patches, see docs/PATCHES.md) plus complete environment recipes, a smoke-test matrix, an 80 GB-server runbook, and the measured outputs of our runs.

• Environment migration works: all 5 stages import- and GPU-verified on py3.11 / torch 2.7.0+cu128 (upstream known-good spanned torch 2.7–2.12). No algorithmic changes — only 3 one-line compat patches (docs/PATCHES.md).
• 16 GB is enough for everything except SAM3D: on an RTX 5080 16 GB, SAM3 segmentation (~9 GB), MoGe pointmaps (~10.5 GB), HaWoR hand reconstruction (~10.5 GB), BootsTAPIR tracking (~9.7 GB) and the MuJoCo-Warp MPC retargeting (~14 GB) all pass. SAM3D mesh generation OOMs at 14.6 GiB torch-allocated during diffusion sampling even in fp16 — a genuine capacity limit (upstream spec ≥32 GB). Measured peak on A800: 20.3 GB, so a ≥24 GB card is the practical floor.
• Full pipeline validated end-to-end on the 138-frame whisking demo: SAM3D mesh (render-IoU 0.7102) → Fast-SAM3D guided tracking of all 138 frames (~44 s/frame at 25 samples on A800) → projection + translation/scale optimization (138/138 frames, tracking err mean 0.0275 → 0.0069) → MuJoCo-Warp MPPI retargeting (final object tracking error pos = 0.0327 m, quat = 0.2109 rad — passes the paper criterion E_pos < 0.1 m, E_rot < 0.5 rad).

Retargeted trajectories ship in retargeting/outputs/sharpa/right/whisking_full/0/ (replay: scripts/run_retarget.sh python replay_viser.py --run-dir retargeting/outputs/sharpa/right/whisking_full/0).

reconstruction/   # official: video → hand+object reconstruction & 6-DoF tracking
  modules/        # sam3, sam-3d-objects, Fast-SAM3D, HaWoR, tapnet (vendored per upstream,
                  # weights/binaries removed — see docs/PREP_INVENTORY.md for sources)
retargeting/      # official: reconstruction output → robot-hand trajectory (MuJoCo Warp MPC)
  outputs/        # shipped demo + OUR whisking_full results (trajectory_mjwp.npz, scene.xml)
deployment/       # official: real-robot replay (UR3e + Sharpa) — reference only, assets removed
docs/             # ENV / SMOKE / A800_RUN / PATCHES / PREP_INVENTORY / PROGRESS / COMPARISON
scripts/          # uv-venv wrappers replacing upstream's conda activation
outputs/a800/     # synced SAM3D mesh + full tracking results + logs
outputs/smoke/    # small local smoke evidence

Removed from the published tree (download or regenerate instead): .venv-*, reconstruction/weights/ (209 MB), reconstruction/outputs/ intermediates (2.9 GB), module checkpoints (HaWoR/SAM3D/etc., ~5.7 GB), whisking/all_frames/ (185 MB, ffmpeg -i whisking.mp4 all_frames/%06d.png), deployment/mujoco_replay/assets/ (46 MB robot meshes, fetch from upstream), SAM3/SAM3D demo media.

Everything uses uv (no conda), Python 3.11, torch 2.7.0+cu128 (--index-url https://download.pytorch.org/whl/cu128). One venv per stage — .venv-sam3, .venv-sam3d, .venv-hawor, .venv-tapnet, .venv-retarget — created with uv venv --python 3.11 .venv-<stage>.

The complete recipes (exact pins, --no-deps ordering, CUDA-extension source builds for pytorch3d / kaolin / gsplat / nvdiffrast / diff-gaussian-rasterization / DROID-SLAM / lietorch / torch-scatter, and the pip-assembled nvcc-12.8 toolkit for machines without system CUDA) are in docs/ENV.md, with frozen uv pip freeze output in docs/requirements-.venv-*.txt. This is the hard part of the reproduction — budget time for the source builds, and set TORCH_CUDA_ARCH_LIST for your GPU.

Run stages through the wrappers (they set CUDA_HOME, HF_HUB_OFFLINE, LIDRA_SKIP_INIT, TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD, …):

scripts/run_sam3.sh     python ...   # SAM3 segmentation
scripts/run_sam3d.sh    python ...   # SAM3D mesh / Fast-SAM3D tracking / stages 3-4
scripts/run_hawor.sh    python ...   # HaWoR hand reconstruction
scripts/run_tapnet.sh   python ...   # BootsTAPIR velocity tracking
scripts/run_retarget.sh python ...   # retargeting (MuJoCo Warp MPC)

Input data: only a monocular RGB video. The shipped reconstruction/whisking/ demo (video + whisk mesh + HaWoR output) plus our synced outputs/a800/ results let you replay/inspect every downstream stage without downloading any model.

Model weights are not redistributed. Sources (details + sizes: docs/PREP_INVENTORY.md, staging layout: docs/ENV.md §Weights):

• Zero-download demo: replay the shipped + our retargeted trajectories — scripts/run_retarget.sh python replay_viser.py --run-dir retargeting/outputs/sharpa/right/whisking_full/0 (needs only .venv-retarget).
• Local smoke: follow docs/SMOKE.md stage by stage (16 GB GPU sufficient except SAM3D).
• SAM3D stages on a big card: docs/A800_RUN.md — venv-clone setup, sm-arch extension rebuild, exact commands, VRAM polls, and the kill/resume/merge procedure for long tracking runs.
• Full pipeline: map upstream reconstruction/run_pipeline.sh stage invocations to the scripts/run_*.sh wrappers (it is conda-based); then retargeting/launch.py --task <name> --raw-dir <reconstruction-output-dir>.

• Deviation from upstream envs: single stack py3.11 / torch 2.7.0+cu128 vs upstream's py3.10–3.12 / torch 2.7–2.12 per-env spread. Validated functionally on the demo, not on the paper's full dataset/scale.
• Deployment not validated (no robot; Sharpa SDK is proprietary).
• The full 138-frame tracking was SIGKILLed mid-run by a neighboring job on the shared server; frames 42–124 of the merged layout use best-sample diffusion poses without the final post-optimization (~2 cm / ~2° less refined, measured on the bounded run) — see docs/A800_RUN.md.
• Local .venv-sam3d pytorch3d ended up CPU-only on the 16 GB machine — run run_project_mesh_combined.py --device cpu there (the A800 clone has the CUDA build).
• Known upstream bugs we hit and documented (workarounds in docs/PATCHES.md): --output-root-dir not threaded through retargeting stages 2–4; no resume logic in track_object.py; ffmpeg frame-offset assumption in the projection script.
• One demo sequence (whisking), not the paper's full evaluation — this study validates the pipeline mechanically; it does not re-measure paper-scale claims.

@article{doasido2026,
  title   = {Do as I Do: Dexterous Manipulation Data from Everyday Human Videos},
  author  = {{Malik Group, UC Berkeley}},
  journal = {arXiv preprint arXiv:2606.19333},
  year    = {2026}
}

(see the arXiv page for the full author list)

All credit for the pipeline and code goes to the original authors of malik-group/do-as-i-do; upstream LICENSE is included. Vendored components keep their own licenses — notably HaWoR: CC BY-NC-ND 4.0 (non-commercial), SAM3 / SAM-3D-Objects: Meta SAM License, tapnet: Apache-2.0 (full table: docs/PREP_INVENTORY.md). Our added docs/scripts/patches may be used under the upstream license terms. This repository is an independent reproduction study and is not affiliated with the authors.

• physercoe/trex-repro — T-Rex tactile-reactive VLA
• physercoe/egoengine-repro — EgoEngine reimplementation
• 3-way method comparison: docs/COMPARISON.md