An independent, from-scratch reproduction of EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations (arXiv:2606.12604). The authors released no official code; everything in this repo is a reimplementation built from the paper text, in simulation only (Isaac Sim 5.1 / IsaacLab 2.3.2), on the TACO dataset.

• Paper: https://arxiv.org/abs/2606.12604
• Artifacts (checkpoints + retargeted motions): https://huggingface.co/physer/egoengine-repro-artifacts
• Eval dataset: TACO (arXiv:2401.08399, project page, instructions repo, CC BY 4.0)

An engineering study that rebuilds the full EgoEngine pipeline stage by stage and reports honestly what worked and what did not — including negative results, which are the point of a reproduction. Scope is sim-only (no real-robot work), on the paper's 16 TACO demonstration triplets.

Pipeline (each stage has a dedicated doc in docs/):

1. TACO loader (docs/TACO_LOADER.md) — hand/object/camera poses + MANO fitting + pseudo-base heuristic, validated on all 16 triplets.
2. Retargeting (docs/RETARGET.md) — mink IK from human hand keypoints to a bimanual Inspire RH56 embodiment, done for 16/16 sequences.
3. Sim scene + refinement (docs/SIM_SCENE.md, docs/REFINE.md) — IsaacLab scene, position-target control, chunked Replay → MPC escalation over all 16 sequences full-frame.
4. RL residual (docs/RL.md, docs/A800_RL.md) — residual PPO (rsl_rl) on the contact-rich chunks that replay/MPC could not fix.
5. Visual branch (docs/VISUAL.md) — SAM2 hand masks → LaMa inpainting → pyrender robot compositing (2-pass blend) to synthesize robot observations from the egocentric video.
6. Policy (docs/POLICY.md) — Appendix-D flow-matching action decoder + HPT-style transformer policy (12.6M params), trained on the generated observations + actions, with a bounded closed-loop sim eval.

The most candid summary: the chunked replay→MPC→RL escalation design is confirmed to be cheap and mostly sufficient (most chunks replay fine; only contact-rich transitions need escalation), and policy learning from purely generated observations works at proof level — data scale, not the pipeline, is the binding constraint.

Evidence artifacts live in this repo under outputs/ (retarget reports, refine summary, RL eval JSONs/plots, policy logs/eval JSONs, smoke matrix). Large artifacts (checkpoints, retargeted trajectories) are on the HF artifacts repo (below).

src/egoengine/    our package: taco/ (loader+MANO+pseudo-base), retarget/ (mink IK),
                  sim/ (IsaacLab scene + replay/MPC), rl/ (residual PPO env),
                  visual/ (SAM2+LaMa+pyrender), policy/ (flow-matching decoder, HPT-style policy)
scripts/          runnable entry points per stage (smoke_*.py, run_retarget.py,
                  run_refine.py, train_residual.py, run_visual.py, train_policy.py,
                  eval_policy.py, run_rl_a800.sh, ...)
docs/             per-stage write-ups + ENV.md (venv recipes), DATA.md, SMOKE.md,
                  PROGRESS.md (chronological work log), COMPARISON.md (3-way repro comparison),
                  requirements-.venv-*.txt (frozen pins)
assets/           Inspire RH56 hand URDF + meshes (see Attribution)
data/             smoke-test input frame (datasets are NOT in the repo — see below)
outputs/          small key result artifacts (evidence for the table above)

Package manager: uv only. Per-stage venvs (dependency sets conflict across stages — do not share one venv):

• Standard stack: Python 3.11, torch 2.7.0+cu128 (https://download.pytorch.org/whl/cu128).
• Sim stages: Isaac Sim 5.1.0.0 + IsaacLab 2.3.2 (= repo-internal 0.54.2), rsl-rl-lib 3.0.1.

Exact recipes, pins, patches and build gotchas: docs/ENV.md, with frozen environments in docs/requirements-.venv-*.txt. Setup scripts: scripts/setup_venv_{sim,recon,policy}*.sh. Third-party repos are referenced (not vendored): see docs/COMPONENTS.md for the exact upstream URLs — SAM2, LaMa / Inpaint-Anything, mink, MoGe, FoundationPose, rsl_rl, HPT, TACO-Instructions.

• TACO dataset (CC BY 4.0): download from the official sources listed in TACO-Instructions; the selective per-triplet download procedure we used is documented in docs/DATA.md and docs/COMPONENTS.md. Expected layout: data/taco -> <TACO root> (the 16 triplets are listed in docs/DATA.md).
• Public model weights (SAM2.1, MANO, HaMeR, big-Lama, MoGe): download from their official sources; our inventory with URLs and expected paths (data/models/...) is in docs/DATA.md / docs/COMPONENTS.md.
• Our artifacts — https://huggingface.co/physer/egoengine-repro-artifacts:
  • policy-checkpoints/ — final flow-matching policy checkpoints (best.pt, last.pt, TorchScript export best_ts.pt + dataset stats)
  • retarget-outputs/ — retargeted bimanual motions for all 16 TACO sequences (trajectory_kinematic.npz + per-sequence reports)
  • rl-checkpoints/ — residual-PPO checkpoints + full training/eval logs

Smoke first (pattern: run, grep SMOKE_OK, never trust Isaac exit codes):

scripts/_smoke_run.sh .venv-sim/bin/python scripts/smoke_isaac_boot.py
# full matrix: docs/SMOKE.md; results: outputs/smoke/results.tsv

Per stage (details + all flags in the stage docs):

# 1. TACO loader validation
.venv-recon/bin/python scripts/validate_taco_loader.py

# 2. mink IK retargeting (one sequence or all 16)
.venv-recon/bin/python scripts/run_retarget.py "(cut, knife, plate)" 20230926_040

# 3. IsaacLab refine (replay / mpc) — requires Isaac Sim venv
OMNI_KIT_ACCEPT_EULA=YES .venv-sim/bin/python scripts/run_refine.py \
  "(cut, knife, plate)" 20230926_040 --mode replay --frames 0:60

# 4. RL residual PPO on an escalated chunk
OMNI_KIT_ACCEPT_EULA=YES .venv-sim/bin/python scripts/train_residual.py \
  "(cut, knife, plate)" 20230926_040 --chunk 40:60 \
  --dense-after-violation --contact-or --warm-start-mpc --curriculum-start-scale 2.0

# 5. Visual branch (SAM2 + LaMa + pyrender blend)
.venv-recon/bin/python scripts/run_visual.py --help

# 6. Policy: build dataset -> train -> export -> closed-loop eval
PYTHONPATH=src .venv-policy/bin/python -m egoengine.policy.dataset --build
.venv-policy/bin/python scripts/train_policy.py --steps 20000
.venv-policy/bin/python scripts/export_policy.py
OMNI_KIT_ACCEPT_EULA=YES .venv-sim/bin/python scripts/eval_policy.py --rollouts 5

RL-at-scale on a remote GPU server (e.g. 4×A800): runbook in docs/A800_RL.md, batch runner scripts/run_rl_a800.sh.

• No official code exists — this is a reimplementation from the paper text; all design choices not specified in the paper are ours and are documented per stage.
• Sim-only; no real-robot evaluation. Embodiment is a bimanual Inspire RH56 floating-base setup (no RB-Y1 arms, no XHand — no public URDF/MJCF found for the paper's XHand1), 36-DoF bimanual action space instead of the paper's arm+hand actions.
• Object masses/frictions are guessed (docs/SIM_SCENE.md blocker #2); a mass/friction sweep exists behind flags in scripts/train_residual.py.
• Reconstruction from video is sidestepped where TACO GT poses exist (same as the paper's sim route); the in-the-wild reconstruction path (FoundationPose etc.) was env-verified but not run at scale.
• Policy training used 585 samples from 5 sequences (frames capped at 0:120) — orders of magnitude below the paper's ~2,500 sequences; the trained policy memorizes (train SR 0.80, held-out SR 0.00). This is a pipeline proof, not a policy-quality result. Resume steps for a real result: docs/POLICY.md §4.
• RL residual stage: delivered but under-budget; the canonical knife-slip chunk was not fixed (docs/RL.md). Closed-loop eval observations keep scene objects at GT pose in the composite (hands rendered at sim state) — see docs/POLICY.md §5.
• 97 refine chunks remain pending_rl (docs/REFINE.md §7).

@article{egoengine2026,
  title  = {EgoEngine: From Egocentric Human Videos to High-Fidelity
            Dexterous Robot Demonstrations},
  author = {Liu, Yangcen and others},
  journal= {arXiv:2606.12604},
  year   = {2026}
}

If you use the TACO data, also cite TACO (arXiv:2401.08399).

• assets/inspire_hand* — Inspire RH56 hand URDF/meshes, derived from the vendor's STEP files via the dex-urdf authors; see assets/inspire_hand_dexurdf/LICENSE.txt.
• TACO dataset: CC BY 4.0, © the TACO authors.
• Third-party projects referenced in docs/COMPONENTS.md (SAM2, LaMa, mink, MoGe, FoundationPose, rsl_rl, HPT) retain their respective licenses.
• Our code in src/ and scripts/ is released under MIT (see LICENSE).

From the same workspace (3-way comparison in docs/COMPARISON.md):

• physercoe/trex-repro — T-Rex tactile-reactive VLA (arXiv:2606.17055)
• physercoe/doasido-repro — Do as I Do (arXiv:2606.19333)