LIBERO-Plus
Robustness benchmark that re-instantiates the LIBERO tasks under seven controlled perturbation factors — camera viewpoint, robot initial state, object layout, language, lighting, background texture and sensor noise — across 10,030 evaluation tasks, plus a 14,347-episode training set.
| Year | 2025 |
|---|---|
| Episodes | 14,347 |
| Total hours | 31 |
| Embodiments | franka_panda |
| Modalities | ee_pose, language, proprioception, rgb |
| Task categories | manipulation |
| License | MIT |
| Access | open — commercial use permitted |
| Maintainer | Fudan University and Shanghai Innovation Institute (with Tongji University and NUS) |
LIBERO-Plus takes the position that headline success rates on LIBERO hide how brittle vision-language-action policies actually are, and it tests that by perturbing the benchmark rather than extending it. The tasks are the same; what changes is everything around them.
Seven perturbation factors, decomposed into 21 low-level components: object layout, camera viewpoints, robot initial states, language instructions, lighting conditions, background textures, and sensor noise. Tasks are stratified into five difficulty levels (L1–L5) based on how a panel of models performs, and the mapping from task ID to perturbation category and level ships with the code.
What the perturbations expose. The authors report success rates falling from about 95% to below 30% under modest changes, with camera viewpoint and robot initial state the most damaging axes. The finding that has attracted the most attention is the opposite one: models are largely insensitive to language perturbation, which the paper attributes to policies substantially ignoring the instruction rather than being robust to how it is phrased. Combined perturbations interact rather than compose independently.
What ships. Two distinct artefacts, which is worth keeping straight:
- 10,030 evaluation tasks — environments and task definitions, not demonstrations. This is the number quoted as the benchmark's size, and it is what `num_tasks` records here. Evaluation runs one trial per task rather than LIBERO's 50.
- A training set of 14,347 episodes / 2,238,036 frames (31.1 h at 20 Hz) spanning 40 task instructions, released in both RLDS (70.4 GiB) and LeRobot v2.1 form. Front and wrist RGB at 256×256, Franka Panda, 20 Hz.
Note the paper describes the fine-tuning corpus as "over 20,000 successful trajectories" while the released LeRobot and RLDS sets contain 14,347 episodes; the released figure is the one recorded here.
Practical note. The evaluation harness is a drop-in replacement — the authors state you can `pip install -e .` over your existing `libero` package without changing evaluation code, adjusting only `num_trials_per_task` from 50 to 1.