Splat2Act：从捕获场景到个性化视觉语言动作（VLA）操作
Splat2Act: From Captured Scenes to Personalized VLA Manipulation
AI总结：
Splat2Act是一种自动化实境到仿真再到实境流水线，用手机扫描和用户物品图像个性化VLA策略，在101个校园环境中生成大量演示，使π₀.₅在真实Stretch 3上拾取成功率达63.9%。
摘要：
将通用操作策略带入家庭或办公室的用户，希望其能从书架、台面和桌子上拾取物品。然而，即使是像π₀.₅这样强大的通用策略，在我们捕获的部署场景中未经过适配时，拾取成功率为0%，且在每个用户环境中收集真实演示的成本高得令人望而却步。我们提出Splat2Act，这是一种自动化的实境到仿真再到实境的流水线，仅使用手持手机扫描和用户物品的图像，就能将预训练的视觉语言动作（VLA）策略个性化到特定场景和物品集合。与之前重建单个台面的实境到仿真工作不同，Splat2Act适用于建筑规模的室内环境：它通过3D高斯溅射构建具有碰撞几何的 photorealistic 数字孪生，自动发现台面和非台面容器，构建可行的拾取任务，并使用特权仿真预言机生成成功的演示，仅需人类捕获场景和物品。应用于101个扫描的校园环境（中位可导航面积为104.87平方米，7个容器），该流水线产生852个容器和超过68000个拾取演示。我们分两个阶段微调π₀.₅：第一阶段适配数百个公共物品资产，这些资产要么仅来自部署场景，要么来自所有101个场景；第二阶段个性化到用户在部署场景容器上的自有物品。我们最强的个性化策略在目标场景中达到62.5%的总拾取成功率。部署在真实的Stretch 3上，同一策略在12个任务子集上达到63.9%的成功率，与这些任务上66.7%的仿真成功率接近。这些结果表明，自动化的实境到仿真再到实境训练可以仅使用仿真预言机数据来个性化VLA操作策略。
英文摘要：
A user who brings a generalist manipulation policy into their home or office wants it to pick up their objects from their shelves, counters, and tables. Yet even a strong generalist like $π_{0.5}$ achieves 0% pickup success in our captured deployment scenes without adaptation, and collecting real demonstrations in every user's environment is prohibitively expensive. We present Splat2Act, an automated real-to-sim-to-real pipeline that personalizes a pretrained vision-language-action (VLA) policy to a specific scene and object set using only a handheld phone scan and images of the user's objects. Unlike prior real-to-sim work that reconstructs individual tabletops, Splat2Act operates on building-scale indoor environments: it builds a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle, with only scene and object capture required from a human. Applied to 101 scanned campus environments with a median navigable area of 104.87 m$^2$ and 7 receptacles, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. We fine-tune $π_{0.5}$ in two stages: Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes; Stage 2 personalizes to the user's own objects on the deployment scene's receptacles. Our strongest personalized policy reaches 62.5% aggregate pickup success in the target scene. Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks. These results suggest that automated real-to-sim-to-real training can personalize VLA manipulation policies using only simulation-oracle data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
