Spatial-UMI-Dream
Contents

Spatial-UMI-Dream:Scaling World Action Models with Metric Geometry
for UMI Manipulation

Weiduo Yuan1,*Ze Yuan1,*Sihan Liu2,*Jionghao Wang3Liang Heng1Zhenyu Zhao1Yue Wang1
1 USC Physical Superintelligence Lab2 CMU3 Texas A&M University

* Equal contribution

PaperarXivSpatial-UMI-DreamRoboTwin-UMI
Spatial-UMI-Dream overview: RGB-D UMI training, spatial world-action modeling, real-world evaluation and RoboTwin-UMI

Overview

UMI collects demonstrations without tying them to a specific robot. But its moving, close-range gripper cameras entangle camera and object motion with no stable external view, leaving the metric geometry needed for precise manipulation implicit in RGB.

Spatial-UMI-Dream is a 5B spatial World Action Model pretrained on 605 hours of RGB-D UMI data to jointly predict future RGB, metric depth, and actions. RoboTwin-UMI is a twenty-task simulation benchmark for evaluating UMI policies under embodiment and environment shifts.

Contributions

Spatial-UMI-Dream

RoboTwin-UMI benchmark

Two robot embodiments share the same UMI gripper and fisheye cameras to test cross-embodiment transfer; randomized lighting on the same tasks tests environmental robustness.

Benchmark

RoboTwin-UMI benchmark construction: original setup, UMI grippers and wrist-mounted fisheye cameras on Aloha-AgileX and ARX-X5, and randomized lighting
RoboTwin-UMI benchmark construction. Left to right: original setup; UMI grippers and wrist fisheye cameras on both embodiments; randomized lighting.

Generalization evaluation protocol

All models train on the twenty tasks with 50 clean Aloha-AgileX demonstrations each, then are evaluated on two embodiments and two lighting conditions without further fine-tuning.

SettingRobotLightingEvaluates
In-domainAloha-AgileXCleanTraining conditions
Cross-embodimentARX-X5CleanEmbodiment transfer
Environment shiftAloha-AgileXRandomizedEnvironmental robustness
Joint shiftsARX-X5RandomizedBoth changes together

Experiments

Simulation results

Stack Blocks TwoAloha · clean lighting · 1×
Mean success / 20 tasks79.10%

Strong in-domain performance.

Mean success rate (%) over twenty tasks

Where the gains come from

Ablation studies

The role of UMI pretraining and metric depth

Four variants share identical downstream demonstrations. The RGB-only variant duplicates its RGB views in place of depth panels, so visual token count and training budget stay matched.

VariantUMI world adaptationJoint world-actionMetric depthIn-domain SR
Wan2.2 (no UMI pretraining)✗✗✓39.50
Spatial-UMI-Dream-Video-only✓✗✓61.25
UMI-Dream (RGB-only)✓✓✗73.35
Spatial-UMI-Dream✓✓✓79.10

Both stages contribute.

UMI world adaptation builds a spatial foundation. Joint world-action pretraining then connects that representation to manipulation.

Metric depth matters.

In-domain success improves over matched RGB-only pretraining, with demonstrations, training budget, and visual token count held fixed.

BibTeX

@misc{yuan2026spatialumidreamscalingworldaction,
  title         = {Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation},
  author        = {Weiduo Yuan and Ze Yuan and Sihan Liu and Jionghao Wang and Liang Heng and Zhenyu Zhao and Yue Wang},
  year          = {2026},
  eprint        = {2611.07229},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2611.07229}
}
Expanded model architecture