发表机构
Viterbi School of Engineering, University of Southern California（南加州大学维特比工程学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
ExStereo通过显式立体表示增强二维VLA模型的三维感知，利用立体匹配和交叉注意力机制，在仿真和真实机器人上显著提升操作性能。
AI 中文摘要
三维感知对于机器人操作至关重要，尤其是在高精度任务中，因为从单目RGB观测中恢复度量深度和精确的三维物体位置本质上是不适定问题。然而，许多视觉-语言-动作（VLA）模型仅依赖RGB观测进行感知。利用立体匹配基础模型的最新进展，我们提出了ExStereo，一个为预训练二维VLA增强三维感知的立体模块。ExStereo从立体图像对重建场景几何，并将多视角观测渲染为显式立体表示，用于立体特征提取。动作专家的动作令牌通过我们提出的动作-立体交叉注意力机制选择性关注生成的立体令牌，使策略能够基于三维场景信息生成机器人动作。为学习鲁棒的三维表示，我们在任务特定后训练之前引入了一个中间训练阶段，使用大规模立体数据上的自监督学习目标。我们通过微调两个公开可用的VLA模型π0.5和SmolVLA来验证我们的方法，并在仿真和真实世界的双臂PiPER平台上进行评估。在两种设置下，使用ExStereo微调的VLA模型始终优于基线，证明了立体感知对机器人操作的有效性。我们的项目网站位于：此https URL。
英文摘要
Three-dimensional perception is critical for robotic manipulation, particularly for high-precision tasks, as recovering metric depth and precise 3D object positions from monocular RGB observations is inherently ill-posed. However, many Vision-Language-Action (VLA) models rely solely on RGB observations for perception. Leveraging recent advances in foundation models for stereo matching, we introduce ExStereo, a stereo module that augments pre-trained 2D VLAs with 3D perception. ExStereo reconstructs scene geometry from stereo image pairs and renders multi-view observations as an explicit stereo representation for stereo feature extraction. The action tokens from the action expert selectively attend to the resulting stereo tokens through our proposed action-stereo cross-attention mechanism, enabling the policy to generate robot actions conditioned on 3D scene information. To learn robust 3D representations, we introduce a mid-training stage before task-specific post-training, using a self-supervised learning objective on large-scale stereo data. We validate our approach by fine-tuning two publicly available VLAs, $π_{0.5}$ and SmolVLA, and evaluate them in simulation and on a real-world bimanual PiPER platform. Across both settings, VLAs fine-tuned with ExStereo consistently outperform baselines, demonstrating the effectiveness of stereo perception for robotic manipulation. Our project website is at: https://exstereo-vla.github.io/ExStereo/.
