LEAP：使特权几何监督对视觉运动学习有效

中文摘要 AI 辅助

特权3D监督在训练期间使用额外的几何信息来引导基于RGB的视觉运动策略学习，而无需在部署时要求几何输入。然而，低重建误差并不能确保视觉表征捕获对控制有用的几何信息。我们识别出削弱这种监督的三个局限性：本体感受捷径、主导视角依赖以及被任务无关几何主导的重建目标。为解决这些局限性，我们提出对齐特权几何的潜在编码（LEAP）。我们的框架使用辅助解码器仅从视觉特征重建操作工作空间内的点云，同时保留本体感受用于动作预测。在完整重建之外，我们引入腕部视角丢弃和与保留腕部匹配的部分重建目标，鼓励编码器捕获互补的局部几何。辅助解码器在推理时被移除，仅留下RGB观测和机器人状态作为策略输入。在RoboTwin、ManiSkill和真实世界任务上的大量实验表明，与扩散策略和ACT相比，LEAP取得了持续且显著的改进，仅参数数量小幅增加且推理速度无下降。

英文摘要

Privileged 3D supervision uses additional geometric information during training to guide RGB-based visuomotor policy learning, without requiring geometric inputs at deployment. However, low reconstruction error does not ensure that visual representations capture geometry useful for control. We identify three limitations that weaken this supervision: proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry. To address these limitations, we propose Latent Encoding with Aligned Privileged Geometry (LEAP). Our framework uses an auxiliary decoder to reconstruct point clouds within the manipulation workspace from visual features alone, while retaining proprioception for action prediction. Alongside full reconstruction, we introduce wrist-view dropout and partial reconstruction targets matched to the retained wrist, encouraging the encoder to capture complementary local geometry. The auxiliary decoder is removed at inference, leaving only RGB observations and robot state as policy inputs. Extensive experiments on RoboTwin, ManiSkill, and real-world tasks demonstrate consistent and substantial improvements over Diffusion Policy and ACT, with only a small increase in parameter count and no reduction in inference speed.

发表机构

• Global College, Shanghai Jiao Tong University（上海交通大学全球学院）
• Shanghai Institute of Physical Intelligence and Robotics (PAIR)（上海物理智能与机器人研究所）
• East China University of Science and Technology（华东理工大学）
• Duke Kunshan University（昆山杜克大学）
• School of Mechanical Engineering, Shanghai Jiao Tong University（上海交通大学机械工程学院）

机构由 AI 辅助整理，请以论文原文为准。