ArmnetBench v0.1 Dataset

3,718 teleoperation and evaluation episodes across 12 different manipulation tasks

ArmnetBench v0.1 is a real-world manipulation benchmark: 7 policies run on a fleet of low-cost SO-101 cells, with every episode human-scored. It is released as evaluation data rather than a training corpus, which makes it unusual here — most of the catalog is demonstrations, and this is what happens when policies actually try.

What is in it. 3,718 episodes across 12 tasks and two hardware configurations: 2,499 episodes on a single SO-101 arm (8 tasks) and 1,219 on a bimanual SO-101 cell (4 tasks). Three cameras per cell at 20 Hz — front/top/wrist for the single arm, top/left-wrist/right-wrist for the bimanual. LeRobot v3.0 format.

Three-way quality labels. Every episode carries `success_class` ∈ {successful, suboptimal, failure} plus a `policy_type` and `policy_repo_id`. 600 episodes are teleoperated reference demonstrations (successful by construction, 50 per task); the remaining 3,118 are autonomous policy rollouts, human-scored. Scoring is strict: only `successful` counts, so the 106 `suboptimal` episodes are recorded as failures. Note that the suboptimal label is only actually populated on 4 of the 12 tasks, so it is not a benchmark-wide partial-credit signal.

Success rates counted from the release, superseded checkpoints excluded (see below):

"Task-balanced" weights each of the 12 tasks equally; the pooled column weights each episode equally, which over-counts the tasks that were run twice. Only ACT and SmolVLA move materially between the two.

Two checkpoint families were excluded, and you should know which. MolmoAct2 was evaluated twice per task. The first family (`molmoact2_<task>_20k`, 20k steps, MEAN_STD normalization) scored 6.7% over 360 episodes; the second (`molmoact2_<task>_quantile_norm_fix_25k`) scored 18.9%. The table above uses only the second. Two caveats: that run changes two variables at once — quantile normalization and 50% more training — so the gain is not attributable to the normalization fix alone; and despite its name its `train_config.json` reports 30,000 steps, not 25,000. ACT similarly has two families, but there they are one lineage rather than two attempts: `act_<task>_20k` declares `pretrained_path: pravsels/act_<task>`, so the un-suffixed repos are the predecessor checkpoint it was warm-started from. Their 60 rollouts (1.7%, 2 tasks only) are excluded rather than averaged in. Pooling both families is what produces the 12.8% and 14.6% figures quoted elsewhere for MolmoAct2 and ACT.

The rollout horizon was not held constant across policies, and this is a real confound. Maximum episode length varies by up to 3.0× between policies on the same task. The sharpest case is GR00T N1.7 on "insert the missing tool": its longest of 30 rollouts is 361 frames (18s at 20 Hz), while the median human demonstration of that task runs 385 frames and the longest runs 690 — 62% of the reference demos would not have fit inside the budget GR00T was given. Its episode lengths cluster at that ceiling (21 of 30 within 5% of the maximum), which is what truncation looks like rather than early failure; π0.5 on the same task spreads from 218 to 667 frames. Averaged across the 12 tasks, the share of human demonstrations longer than the policy's own horizon is 9.5% for GR00T and 5.0% for Diffusion Policy, against 0.8% for π0.5. The two policies with the tightest budgets are penalised most, and the leader was the least constrained.

Read this table as checkpoint scores, not a policy ranking. Each entry is one specific released checkpoint, trained on 50 demonstrations per task and evaluated on 30 rollouts per task (45–60 on the handful run twice), under a single harness, by three different authors using three different training pipelines. The 50-demo budget is shared and deliberately small. At n=30 per cell the binomial standard error is 6–9pp depending on the rate, so a 95% interval on any single per-task number spans roughly ±13–18pp: per-task figures separate the extremes, not neighbours. None of these figures should be read as what the policy family achieves when tuned.

Tasks. Single-arm: insert a ring onto a peg, insert/remove a tool from a toolbox, clip and unclip a DisplayPort cable, move eye drops to a basket and to a shelf, stack blocks. Bimanual: fold a tea towel, hold a lamp steady with one arm while opening a door with the other, insert a candle into a lantern and close the door, transfer a cube between arms into a basket.

A note on the numbers. The paper reports 3,118 core episodes (2,518 policy rollouts + 600 demonstrations). The released repos contain 3,718 episodes — 3,118 rollouts plus the 600 dem