Stanford HomeBody Humanoid Uses SLAM and Isaac Sim for Long-Horizon Loco-Manipulation

The system first directs the humanoid to explore an unseen kitchen while recording ego-view imagery, depth, SLAM geometry, and Astra-selected waypoints. These data streams feed a reconstruction pipeline that produces a geometrically and semantically consistent Isaac Sim twin. The VLM then queries this persistent spatial model to select targets and sequence skills when objects lie outside the current camera frustum.

Demonstrated behaviors include gathering coffee bags onto an island, discarding spoiled cartons, retrieving medicine from an initially occluded drawer, and coordinating bimanual actions during repeated traversals. Each skill returns execution feedback that triggers local retries or VLM replanning. The skill library shares a uniform target-and-result interface, enabling composition without task-specific scripts.

Prior coverage understates the dependence on SLAM loop closure accuracy and the sim-to-real texture gap that can degrade VLM grounding after viewpoint changes. The approach also omits quantitative success rates or failure modes under lighting variation and dynamic obstacles. Operationally, the pipeline trades exploration time for reduced need for human-provided maps or fiducials.

Next steps include scaling the skill set to additional articulated objects and testing closed-loop transfer on physical hardware in novel kitchens.