EgoInfinity
Web-scale data engine that automatically lifts in-the-wild human video into metric 4D hand-object interaction and retargets it onto four robot embodiments; 142M source clips, with a 1,150-clip open preview.
| Year | 2026 |
|---|---|
| Episodes | 1,150 |
| Embodiments | franka_panda, human, unitree_humanoid |
| Modalities | depth, ee_pose, language, segmentation, rgb |
| Task categories | manipulation, perception |
| License | custom |
| Access | open |
| Maintainer | Rice RobotPI Lab |
EgoInfinity is not a static dataset but a modular, upgradable data engine: it takes ordinary internet video of people manipulating objects and automatically recovers metric 4D hand–object interaction, then retargets that motion into executable robot joint trajectories — no manual annotation and no depth sensors.
- Runs over Meta FAIR's Action100M corpus — 14.6 years (~127,000 h) of footage across ~142M action clips. The publicly released artifact is a curated preview subset of 1,150 rows / 104 fully retargeted clips (5.49 GB).
- Eight-stage pipeline: metric depth + focal estimation (MoGe-2), gravity calibration (GeoCalib), hand detection and MANO reconstruction (YOLO + WiLoR), optical flow and depth stabilisation (MEMFOF), motion infilling (HaWoR), text-prompted detection and tracking (SAM 3.1 + SAM 2), single-image 3D object reconstruction, and 6-DoF mesh pose tracking.
- Each processed clip yields per-frame 21-joint hand poses, MANO meshes, per-object 6-DoF pose trajectories, Gaussian-splat object meshes, optical flow, metric depth, and segmentation masks.
- Retargets to four embodiments: Franka Panda (7-DoF + gripper), Unitree G1 (7-DoF arm + 7-DoF hand), NASA Robonaut 2 (7-DoF arm + 18-DoF hand), and XLeRobot (5-DoF + gripper), with per-frame IK-success and manipulability metrics stored alongside.
- Supports arbitrary camera viewpoints and shot sizes, including partial body visibility — unlike prior pipelines that assume a fixed egocentric rig.
- Validated on a real robot for grasping, cutting, wiping, and pouring learned directly from internet video.
- Because it is an engine rather than a fixed corpus, data quality improves automatically as its component models are swapped for better ones.
Licensing is non-commercial research only (FAIR Noncommercial Research License v1 for data; the code carries mixed MIT / CC-BY-NC-ND / AGPL components).