SO-101 Bench

Accessible Spatial Manipulation

Truman Hickok · Independent · hickoktruman@gmail.com

Low-cost arms like the SO-101 are now the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box. SO-101 Bench is a simulated benchmark built on a digital twin of a real SO-101 setup: four tasks, 138 objects in three generalization splits and 916 evaluation episodes. Policies trained only on its simulated data keep their ranking on the real robot, and run out of the box in unseen real environments with unseen objects.

SO-101 Bench at a glance: (a) the real SO-101 and its Isaac Lab digital twin with objects from each split; (b) the four simulated tasks; (c) five policies, including an LLM agent, keep their exact ranking on the real robot (Spearman ρ = 1.00, MMRV = 0.00); (d) the 916-episode leaderboard, led by GPT-6.1 Sol with PyRUA-Lean and Galaxea 0.5; (e) the harness ablation; (f) the language-model comparison

SO-101 Bench

Four tasks, scored as five families: bin (one or four objects), next to, between, and directional move.

Primary-evaluation demonstrations · Four policies · All successful · 1×
“Place the harmonica between the toy boat and the air pump.”Episode 215 · InternW0-Δ
“Place the flower pot next to the bottle opener.”Episode 123 · NVIDIA GR00T N1.7-3B
“Move the sponge backwards.”Episode 276 · Physical Intelligence π0.5
“Place each object in the plastic bin.”Episode 85 · Galaxea 0.5
Simulation objects by generalization split
48 seen objects, 43 unseen instances of seen classes, and 47 unseen classes. The 37 private unseen-class objects are not shown. Orange dots mark physical objects also used on the real robot (39 shown).

Real-Only vs. Sim-Only Policy Correspondence

Independently trained NVIDIA GR00T N1.6-3B policies are compared on 906 episodes matched by task, instruction and ordered object set. Across 15 task × split conditions, Pearson r = 0.86 and Spearman ρ = 0.74.

Real-only versus sim-only policy correspondence
Success by task and object split, failure composition, and four-object bin progress. The policies differ in training data as well as evaluation domain; this comparison does not isolate simulator fidelity.NVIDIA GR00T N1.6-3B (real-only and sim-only).
“Place the clip next to the pen.”
Real robot · real-only policy · 1× wall-clock
Simulation · sim-only policy · 1× simulation time

Sim2Real with No Real Data

Five policies trained only on sim data are evaluated in simulation and on the real robot, on 45 matched layouts. The ranking of all five is preserved. Median episode lengths: 197s Sol, 38s Galaxea 0.5, 38s GR00T N1.7, 59s π0.5 and 56s MolmoAct2-SO101.

Sim2real policy ranking and success by task family

Four-object bin Physical Intelligence π0.5

“Place each object in the plastic bin.”
Real robot · 1× wall-clock
Simulation · 1× simulation time

Matched trial 34 and identical sim-trained weights; independent successful rollouts.

Next to GPT-6.1 Sol + PyRUA-Lean + Galaxea 0.5

“Place the bowl next to the screwdriver.”
Real robot · 5× playback
Simulation · 1× simulation time

Matched trial 22; independent successful rollouts. Real video defaults to 5× playback; simulation defaults to 1×. Real placement was operator-scored as success; the controller logged a timeout.

Between Galaxea 0.5

“Place the flower pot between the tape and the clip.”
Real robot · 1× wall-clock
Simulation · 1× simulator time

Identical sim-trained weights, matched initial layout, independent rollouts. Both episodes succeed.

Directional move NVIDIA GR00T N1.7-3B

“Move the wires right.”
Real robot · 1× wall-clock
Simulation · 1× simulation time

Matched layout and identical sim-trained weights; independent successful rollouts.

Leaderboard

Nine learned policies and two LLM agents on the 916-episode primary evaluation.

916 primary-evaluation episodes span 15 task-family × object-split cells. ROBUST: 178 held-out appearance episodes; PLAN: 48 rotation-to-fit episodes; MEM: 51 occlusion episodes. Subsets overlap.

Learned policies receive 36 hours of fine-tuning on 4 × A100-80GB GPUs; checkpoints are selected on 39 held-out validation episodes.

SO-101 Bench primary-evaluation leaderboard
Score is the mean success over 15 task × split cells with a 95% bootstrap confidence interval. Results compare a fixed fine-tuning compute budget, not a capability ceiling.

Open the live leaderboard ↗

PLAN — Rotate the toy boat to fit between the pill planner and the camera.Episode 529 · GPT-6.1 Sol + PyRUA-Lean + Galaxea 0.5 · Success · All 9 learned policies failed · 1× simulation time
MEM — Move the tape backwards, retaining the boundary object position while it is occluded during the move.Episode 300 · ImageWAM · Success · Solved by 4 of 9 learned policies · 1× simulation time
Solved only by GPT-6.1 Sol + PyRUA-Lean v9.6 + Galaxea 0.5 · Overhead camera · 1×
“Place the hand brush between the white pen and the bowl.”Episode 174 · PLAN · Success · 1× simulation time
“Place the trimmer between the spool and the bow tie.”Episode 471 · PLAN · Success · 1× simulation time
“Place the camera between the die and the whistle.”Episode 469 · Success · 1× simulation time
“Place the bottle opener between the kazoo and the corkscrew.”Episode 509 · Success · 1× simulation time

All ten other leaderboard entries failed each of these episodes. Simulator time pauses while the LLM computes, so the videos run at 1× simulation time.

LLM Agents: Harness and Model Ablations

Ablations use 60 fixed primary-evaluation episodes: 30 easy and 30 hard, selected before running the agents.

Harness ablation on 60 paired episodes
Success of each harness with three LLMs, and of PyRUA-Lean v9.6 + Galaxea 0.5 with a fourth LLM, MiMo v2.6 Pro; successes by task family for GPT-6.1 Sol. Intervals are 95% Wilson; hatched bars indicate runs stopped before 60 episodes. The dashed line marks Galaxea 0.5 alone.Grasping policy: Galaxea 0.5. Language models: GPT-6.1 Sol, Gemini 4 Argon, DeepSeek V4.1 Flash, MiMo v2.6 Pro.

With PyRUA-Lean + Galaxea 0.5 held fixed, success is 85% for GPT-6.1 Sol, 77% for Gemini 4 Argon, 65% for DeepSeek V4.1 Flash and 55% for MiMo v2.6 Pro.

Cost and token usage over 60 paired episodes
LLM (effort)SuccessAPI costOutput tokens (reasoning)Input tokens (cached)
GPT-6.1 Sol (high)85% (51/60)$14.680.32M (0.15M)61.82M (96%)
Gemini 4 Argon (medium)77% (46/60)$74.251.11M (0.86M)42.30M (73%)
DeepSeek V4.1 Flash (high)65% (39/60)$1.401.45M (1.23M)87.33M (98%)
MiMo v2.6 Pro (high)55% (33/60)$2.631.06M (0.82M)75.33M (96%)
Galaxea 0.5 alone53% (32/60)$0——

Output includes reasoning tokens. Costs use list prices (Gemini: gross Vertex rate); Argon used medium reasoning effort, the other LLMs high. Simulator time pauses during LLM computation.

Resources

Code and evaluation protocol ↗ Sim demonstrations ↗ Sim demonstrations with initial frames ↗ Real demonstrations ↗ Digital twin assets ↗

3,629 sim demonstrations (18.5 h); approximately 4,400 real demonstrations (20 h).

Fine-tuned policy checkpoints · 10 models on Hugging Face

Every learned leaderboard policy, fine-tuned on so101_bench_sim under the shared 36 h budget. These are the weights that run out of the box on the real SO-101.

Citation

Download .bib

If you use SO-101 Bench in your research, please cite:

@misc{hickok2026so101bench,
  title  = {{SO-101 Bench: Accessible Spatial Manipulation}},
  author = {Hickok, Truman},
  year   = {2026},
  url    = {https://github.com/5hadytru/so101_bench}
}