arXiv 2606.290892026-06-27cs.RO

TAP-VLA：面向视觉语言动作模型的触觉标注提示

TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models

Robotics Department, University of Michigan（密歇根大学机器人系）；Computer Science and Engineering Department, University of Michigan（密歇根大学计算机科学与工程系）

Mark Van der Merwe, Mohamad Louai Shehab, Jayjun Lee, Youngsun Wi, Yinpei Dai, Dmitry Berenson, Nima Fazeli

备注：: 8 pages + references

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出TAP-VLA框架，通过将触觉剪切场叠加到RGB图像上作为视觉增强，无需修改架构或触觉预训练，即可将触觉反馈融入VLA模型，在接触丰富任务中成功率78%，显著优于基线。

摘要：

视觉-语言-动作（VLA）模型通过利用大规模视觉和语言预训练，在视觉、语义和空间任务变化上展现出令人印象深刻的推理能力。然而，它们对接触力基本无感知，接触力很少在视觉反馈中清晰显现，但对接触丰富的操作至关重要。触觉传感直接测量这些力，但将其集成到VLA中很困难：触觉数据在用于预训练VLA的大规模语料库中不存在，因此将其作为新输入模态会引入分布偏移，侵蚀使VLA有效的预训练。我们提出面向视觉语言动作模型的触觉标注提示（TAP-VLA），一个简单的框架，通过视觉增强而非架构改变来提供触觉反馈。TAP-VLA从视觉-触觉传感器提取剪切场，并将其作为空间接地向量叠加到策略已使用的多视图RGB图像上，在VLA的原生观测空间中产生清晰、可解释的触觉线索。由于架构未改动，该方法无需触觉预训练，计算开销可忽略，且接近预训练分布。在四个接触丰富任务中，TAP-VLA在78%的试验中成功，而仅视觉微调和替代触觉融合基线的成功率低于50%——包括基线表现不比随机好多少的任务。

英文摘要：

Vision-Language-Action (VLA) models demonstrate impressive reasoning over visual, semantic, and spatial task variations by leveraging large-scale vision and language pre-training. They remain, however, largely blind to contact forces, which seldom manifest clearly in visual feedback but are central to contact-rich manipulation. Tactile sensing measures these forces directly, but integrating it into VLAs is difficult: tactile data is absent from the large-scale corpora used to pre-train VLAs, so adding it as a new input modality induces a distribution shift that erodes the very pre-training that makes VLAs effective. We propose Tactile Annotation Prompting for Vision-Language-Action models (TAP-VLA), a simple framework that supplies tactile feedback through visual augmentation rather than architectural change. TAP-VLA extracts shear fields from visuo-tactile sensors and overlays them as spatially-grounded vectors onto the multi-view RGB images the policy already consumes, yielding a clear, interpretable tactile cue in the VLA's native observation space. Because the architecture is untouched, the approach requires no tactile pre-training, adds negligible compute, and stays close to the pre-training distribution. Across four contact-rich tasks, TAP-VLA succeeds on 78% of trials, compared to under 50% for vision-only fine-tuning and alternative tactile-fusion baselines -- including tasks where the baselines perform no better than chance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。