arXiv 2606.172562026-06-17cs.ROcs.CV

对比动作-图像预训练用于视觉运动控制

Contrastive Action-Image Pre-training for Visuomotor Control

UC Berkeley（加州大学伯克利分校）；NVIDIA（英伟达）；Sapienza University of Rome（罗马大学）；Panasonic（松下）；ItalAI

Yuvan Sharma, Dantong Niu, Anirudh Pai, Zekai Wang, Zhuoyang Liu, Baifeng Shi, Stefano Saravalle, Boning Shao, Ruijie Zheng, Jing Wang, Konstantinos Kallidromit… 展开作者

Yuvan Sharma, Dantong Niu, Anirudh Pai, Zekai Wang, Zhuoyang Liu, Baifeng Shi, Stefano Saravalle, Boning Shao, Ruijie Zheng, Jing Wang, Konstantinos Kallidromitis, Yusuke Kato, Fabio Galasso, Yuke Zhu, Danfei Xu, Linxi "Jim" Fan, Jitendra Malik, Trevor Darrell, Roei Herzig

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出CAIP方法，利用大规模第一人称视频中3D手部关键点作为代理动作信号，通过对比学习统一动作-图像表示，在少量机器人数据下显著提升灵巧操作性能。

摘要：

现有的机器人视觉编码器面临一个根本瓶颈：机器人数据集缺乏大规模预训练所需的规模。先前的工作通过转向互联网规模的图像和语言数据或自我中心的人类视频来规避数据稀缺问题。虽然这些模型显示出潜力，但两种范式都没有从配对的视觉和动作数据中学习，而下游视觉运动控制策略需要这些数据。然而，机器人轨迹作为这种配对信号最直接的来源，在预训练规模上不可用，这促使我们从丰富的人类视频中提取动作信号。为此，我们引入了CAIP（对比动作-图像预训练），一种视觉编码器，将大规模自我中心视频中的人类手部姿态视为末端执行器动作的代理。通过提取3D手部关键点（一种与下游机器人动作空间自然对齐的表示），CAIP通过对比目标学习统一的动作-图像表示。利用32,041小时的自我中心人类视频和仅88小时的机器人操作数据，CAIP优于最先进的视觉编码器，包括DINOv2、SigLIP、MVP和R3M。在使用Dexmate Vega和Sharpa Wave手的具有挑战性的真实世界灵巧操作设置上评估，CAIP在涉及折叠、倾倒和精细操作的任务上取得了超过30%的性能提升。我们的结果表明，我们的对比动作中心预训练方法为获得更适合物理交互的鲁棒视觉表示提供了一条可扩展的路径。

英文摘要：

Existing vision encoders for robotics face a fundamental bottleneck: robotic datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image and language data or egocentric human video. While these models show promise, neither paradigm learns from paired vision and action data, which downstream visuomotor control policies require. However, robot trajectories, the most direct source of this paired signal, are not available at pre-training scale, motivating us to extract action signals from abundant human video instead. To this end, we introduce CAIP (Contrastive Action-Image Pre-training), a vision encoder that treats human hand poses from large-scale egocentric video as a proxy for end-effector actions. By extracting 3D hand keypoints, a representation that aligns naturally with downstream robot action spaces, CAIP learns a unified action-image representation through a contrastive objective. Leveraging 32,041 hours of egocentric human video and only 88 hours of robotic manipulation data, CAIP outperforms state-of-the-art vision encoders including DINOv2, SigLIP, MVP, and R3M. Evaluated on a challenging real-world dexterous manipulation setup using Dexmate Vega and Sharpa Wave hands, CAIP yields performance gains of more than 30% on tasks involving folding, pouring, and fine-grained manipulation. Our results show that our method of contrastive action-centric pre-training yields a scalable path to achieving robust visual representations better suited for physical interaction.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。