arXiv 2505.095772025-05-14cs.RO

VTLA：面向插入操作的、带偏好学习的视觉-触觉-语言-动作模型

VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；Samsung R&D Institute China–Beijing（三星中国北京研发中心）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）

Chaofan Zhang, Peng Hao, Xiaoge Cao, Xiaoshuai Hao, Shaowei Cui, Shuo Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出融合视觉、触觉、语言与动作的 VTLA 模型，并利用 DPO 提供连续动作监督，在指尖插入任务中显著优于基线并实现良好 Sim2Real 迁移。

摘要：

尽管视觉-语言模型已取得显著进展，但它们在语言条件化机器人操作中的应用仍未得到充分探索，尤其是在超越以视觉为主的抓放场景、需要丰富接触的任务中。为弥合这一差距，我们提出了 Vision-Tactile-Language-Action 模型，这是一个新颖框架，通过跨模态语言锚定有效整合视觉与触觉输入，从而在接触密集场景中实现鲁棒的策略生成。我们在仿真环境中构建了一个低成本多模态数据集，其中包含专为指尖插入任务设计的视觉-触觉-动作-指令对。此外，我们引入 Direct Preference Optimization（DPO，直接偏好优化），为 VTLA 模型提供类似回归的监督，有效弥合基于分类的下一 token 预测损失与连续机器人任务之间的差距。实验结果表明，VTLA 模型优于传统模仿学习方法（如 diffusion policies）和现有多模态基线（TLA/VLA），在未见过的销钉形状上实现超过90%的成功率。最后，我们开展真实世界的销钉入孔实验，以证明所提出 VTLA 模型出色的 Sim2Real 性能。补充视频和结果请访问项目网站：https://sites.google.com/view/vtla

英文摘要：

While vision-language models have advanced significantly, their application in language-conditioned robotic manipulation is still underexplored, especially for contact-rich tasks that extend beyond visually dominant pick-and-place scenarios. To bridge this gap, we introduce Vision-Tactile-Language-Action model, a novel framework that enables robust policy generation in contact-intensive scenarios by effectively integrating visual and tactile inputs through cross-modal language grounding. A low-cost, multi-modal dataset has been constructed in a simulation environment, containing vision-tactile-action-instruction pairs specifically designed for the fingertip insertion task. Furthermore, we introduce Direct Preference Optimization (DPO) to offer regression-like supervision for the VTLA model, effectively bridging the gap between classification-based next token prediction loss and continuous robotic tasks. Experimental results show that the VTLA model outperforms traditional imitation learning methods (e.g., diffusion policies) and existing multi-modal baselines (TLA/VLA), achieving over 90% success rates on unseen peg shapes. Finally, we conduct real-world peg-in-hole experiments to demonstrate the exceptional Sim2Real performance of the proposed VTLA model. For supplementary videos and results, please visit our project website: https://sites.google.com/view/vtla

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。