Skip to content

README.md

Latest commit

 

History

History
209 lines (170 loc) · 10.7 KB

README.md

File metadata and controls

209 lines (170 loc) · 10.7 KB
Raw

EgoEngine Reproduction (egoengine-repro)

An independent, from-scratch reproduction of EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations (arXiv:2606.12604). The authors released no official code; everything in this repo is a reimplementation built from the paper text, in simulation only (Isaac Sim 5.1 / IsaacLab 2.3.2), on the TACO dataset.

What this is

An engineering study that rebuilds the full EgoEngine pipeline stage by stage and reports honestly what worked and what did not — including negative results, which are the point of a reproduction. Scope is sim-only (no real-robot work), on the paper's 16 TACO demonstration triplets.

Pipeline (each stage has a dedicated doc in docs/):

  1. TACO loader (docs/TACO_LOADER.md) — hand/object/camera poses + MANO fitting + pseudo-base heuristic, validated on all 16 triplets.
  2. Retargeting (docs/RETARGET.md) — mink IK from human hand keypoints to a bimanual Inspire RH56 embodiment, done for 16/16 sequences.
  3. Sim scene + refinement (docs/SIM_SCENE.md, docs/REFINE.md) — IsaacLab scene, position-target control, chunked Replay → MPC escalation over all 16 sequences full-frame.
  4. RL residual (docs/RL.md, docs/A800_RL.md) — residual PPO (rsl_rl) on the contact-rich chunks that replay/MPC could not fix.
  5. Visual branch (docs/VISUAL.md) — SAM2 hand masks → LaMa inpainting → pyrender robot compositing (2-pass blend) to synthesize robot observations from the egocentric video.
  6. Policy (docs/POLICY.md) — Appendix-D flow-matching action decoder + HPT-style transformer policy (12.6M params), trained on the generated observations + actions, with a bounded closed-loop sim eval.

Headline results

stage result
TACO loader 16/16 triplets validated (poses, MANO, pseudo-base)
mink IK retargeting 16/16 sequences retargeted (thumb-limited residuals, see docs/RETARGET.md)
Refine (full-frame, 16 seqs) replay success 8/16, MPC success 8/16; 97 chunks escalated to pending_rl (failure taxonomy in docs/REFINE.md §7). Paper's sim action SR on TACO: 0.83 — scope differs (2 objects/scene, guessed masses, Inspire hands, no arm)
RL residual-PPO stage delivered with a 5-flag improvement bundle (dense reward, contact-OR, MPC warm start, feasibility curriculum, mass/friction sweep); the canonical knife-slip chunk [40,60) was NOT fixed within budget — candid negative result, see docs/RL.md and outputs/rl/**/eval_residual.json
Visual branch SAM2 + LaMa + pyrender 2-pass blend verified end-to-end (~1 s/frame; no video diffusion needed)
Policy (App-D flow matching) trained on 585 generated samples (5 train / 1 held-out seq): train loss 2.05 → 0.54; closed-loop sim eval — 4/5 (0.80) on a train sequence, 0/5 (0.00) held-out: the pipeline closes end-to-end, but the policy memorizes at this data scale. Full-data training NOT done (resume steps in docs/POLICY.md §4)
Smoke matrix 7/7 stages OK on a single RTX 5080 16GB (outputs/smoke/results.tsv)

The most candid summary: the chunked replay→MPC→RL escalation design is confirmed to be cheap and mostly sufficient (most chunks replay fine; only contact-rich transitions need escalation), and policy learning from purely generated observations works at proof level — data scale, not the pipeline, is the binding constraint.

Evidence artifacts live in this repo under outputs/ (retarget reports, refine summary, RL eval JSONs/plots, policy logs/eval JSONs, smoke matrix). Large artifacts (checkpoints, retargeted trajectories) are on the HF artifacts repo (below).

Repository layout

src/egoengine/    our package: taco/ (loader+MANO+pseudo-base), retarget/ (mink IK),
                  sim/ (IsaacLab scene + replay/MPC), rl/ (residual PPO env),
                  visual/ (SAM2+LaMa+pyrender), policy/ (flow-matching decoder, HPT-style policy)
scripts/          runnable entry points per stage (smoke_*.py, run_retarget.py,
                  run_refine.py, train_residual.py, run_visual.py, train_policy.py,
                  eval_policy.py, run_rl_a800.sh, ...)
docs/             per-stage write-ups + ENV.md (venv recipes), DATA.md, SMOKE.md,
                  PROGRESS.md (chronological work log), COMPARISON.md (3-way repro comparison),
                  requirements-.venv-*.txt (frozen pins)
assets/           Inspire RH56 hand URDF + meshes (see Attribution)
data/             smoke-test input frame (datasets are NOT in the repo — see below)
outputs/          small key result artifacts (evidence for the table above)

Installation

Package manager: uv only. Per-stage venvs (dependency sets conflict across stages — do not share one venv):

  • Standard stack: Python 3.11, torch 2.7.0+cu128 (https://download.pytorch.org/whl/cu128).
  • Sim stages: Isaac Sim 5.1.0.0 + IsaacLab 2.3.2 (= repo-internal 0.54.2), rsl-rl-lib 3.0.1.

Exact recipes, pins, patches and build gotchas: docs/ENV.md, with frozen environments in docs/requirements-.venv-*.txt. Setup scripts: scripts/setup_venv_{sim,recon,policy}*.sh. Third-party repos are referenced (not vendored): see docs/COMPONENTS.md for the exact upstream URLs — SAM2, LaMa / Inpaint-Anything, mink, MoGe, FoundationPose, rsl_rl, HPT, TACO-Instructions.

Data & checkpoints

  • TACO dataset (CC BY 4.0): download from the official sources listed in TACO-Instructions; the selective per-triplet download procedure we used is documented in docs/DATA.md and docs/COMPONENTS.md. Expected layout: data/taco -> <TACO root> (the 16 triplets are listed in docs/DATA.md).
  • Public model weights (SAM2.1, MANO, HaMeR, big-Lama, MoGe): download from their official sources; our inventory with URLs and expected paths (data/models/...) is in docs/DATA.md / docs/COMPONENTS.md.
  • Our artifacts — https://huggingface.co/physer/egoengine-repro-artifacts:
    • policy-checkpoints/ — final flow-matching policy checkpoints (best.pt, last.pt, TorchScript export best_ts.pt + dataset stats)
    • retarget-outputs/ — retargeted bimanual motions for all 16 TACO sequences (trajectory_kinematic.npz + per-sequence reports)
    • rl-checkpoints/ — residual-PPO checkpoints + full training/eval logs

Reproduce

Smoke first (pattern: run, grep SMOKE_OK, never trust Isaac exit codes):

scripts/_smoke_run.sh .venv-sim/bin/python scripts/smoke_isaac_boot.py
# full matrix: docs/SMOKE.md; results: outputs/smoke/results.tsv

Per stage (details + all flags in the stage docs):

# 1. TACO loader validation
.venv-recon/bin/python scripts/validate_taco_loader.py

# 2. mink IK retargeting (one sequence or all 16)
.venv-recon/bin/python scripts/run_retarget.py "(cut, knife, plate)" 20230926_040

# 3. IsaacLab refine (replay / mpc) — requires Isaac Sim venv
OMNI_KIT_ACCEPT_EULA=YES .venv-sim/bin/python scripts/run_refine.py \
  "(cut, knife, plate)" 20230926_040 --mode replay --frames 0:60

# 4. RL residual PPO on an escalated chunk
OMNI_KIT_ACCEPT_EULA=YES .venv-sim/bin/python scripts/train_residual.py \
  "(cut, knife, plate)" 20230926_040 --chunk 40:60 \
  --dense-after-violation --contact-or --warm-start-mpc --curriculum-start-scale 2.0

# 5. Visual branch (SAM2 + LaMa + pyrender blend)
.venv-recon/bin/python scripts/run_visual.py --help

# 6. Policy: build dataset -> train -> export -> closed-loop eval
PYTHONPATH=src .venv-policy/bin/python -m egoengine.policy.dataset --build
.venv-policy/bin/python scripts/train_policy.py --steps 20000
.venv-policy/bin/python scripts/export_policy.py
OMNI_KIT_ACCEPT_EULA=YES .venv-sim/bin/python scripts/eval_policy.py --rollouts 5

RL-at-scale on a remote GPU server (e.g. 4×A800): runbook in docs/A800_RL.md, batch runner scripts/run_rl_a800.sh.

Limitations & deviations

  • No official code exists — this is a reimplementation from the paper text; all design choices not specified in the paper are ours and are documented per stage.
  • Sim-only; no real-robot evaluation. Embodiment is a bimanual Inspire RH56 floating-base setup (no RB-Y1 arms, no XHand — no public URDF/MJCF found for the paper's XHand1), 36-DoF bimanual action space instead of the paper's arm+hand actions.
  • Object masses/frictions are guessed (docs/SIM_SCENE.md blocker #2); a mass/friction sweep exists behind flags in scripts/train_residual.py.
  • Reconstruction from video is sidestepped where TACO GT poses exist (same as the paper's sim route); the in-the-wild reconstruction path (FoundationPose etc.) was env-verified but not run at scale.
  • Policy training used 585 samples from 5 sequences (frames capped at 0:120) — orders of magnitude below the paper's ~2,500 sequences; the trained policy memorizes (train SR 0.80, held-out SR 0.00). This is a pipeline proof, not a policy-quality result. Resume steps for a real result: docs/POLICY.md §4.
  • RL residual stage: delivered but under-budget; the canonical knife-slip chunk was not fixed (docs/RL.md). Closed-loop eval observations keep scene objects at GT pose in the composite (hands rendered at sim state) — see docs/POLICY.md §5.
  • 97 refine chunks remain pending_rl (docs/REFINE.md §7).

Citation

@article{egoengine2026,
  title  = {EgoEngine: From Egocentric Human Videos to High-Fidelity
            Dexterous Robot Demonstrations},
  author = {Liu, Yangcen and others},
  journal= {arXiv:2606.12604},
  year   = {2026}
}

If you use the TACO data, also cite TACO (arXiv:2401.08399).

Attribution & license

  • assets/inspire_hand* — Inspire RH56 hand URDF/meshes, derived from the vendor's STEP files via the dex-urdf authors; see assets/inspire_hand_dexurdf/LICENSE.txt.
  • TACO dataset: CC BY 4.0, © the TACO authors.
  • Third-party projects referenced in docs/COMPONENTS.md (SAM2, LaMa, mink, MoGe, FoundationPose, rsl_rl, HPT) retain their respective licenses.
  • Our code in src/ and scripts/ is released under MIT (see LICENSE).

Related reproductions

From the same workspace (3-way comparison in docs/COMPARISON.md):