OmniVTLA：具有语义对齐触觉感知的视觉-触觉-语言-动作模型
OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensing
AI总结：
提出OmniVTLA架构，通过双路径触觉编码器框架和语义对齐触觉ViT，结合新数据集ObjTac，显著提升机器人操作中接触密集任务的成功率和轨迹平滑度。
摘要：
最近的视觉-语言-动作（VLA）模型建立在视觉-语言基础之上，在机器人操作中取得了有希望的结果并展现出任务泛化的可能性。然而，由于触觉传感器的异质性和获取触觉数据的困难，当前的VLA模型显著忽视了触觉感知的重要性，并在接触密集任务中失败。为了解决这个问题，本文提出了OmniVTLA，一种涉及触觉感知的新型架构。具体来说，我们的贡献有三方面。首先，我们的OmniVTLA具有双路径触觉编码器框架。该框架通过使用预训练的视觉变换器（ViT）和语义对齐的触觉ViT（SA-ViT），增强了跨多种基于视觉和基于力的触觉传感器的触觉感知。其次，我们引入了ObjTac，一个全面的基于力的触觉数据集，捕捉了10个类别56个物体的文本、视觉和触觉信息。拥有135K三模态样本，ObjTac补充了现有的视觉-触觉数据集。第三，利用该数据集，我们训练了一个语义对齐的触觉编码器，以学习统一的触觉表示，作为OmniVTLA更好的初始化。真实世界实验表明，与最先进的VLA基线相比，取得了显著改进，在拾取和放置任务中，使用夹爪的成功率达到96.9%（比基线高21.9%），使用灵巧手的成功率达到100%（比基线高6.2%）。此外，与现有VLA相比，OmniVTLA通过触觉感知显著减少了任务完成时间并生成了更平滑的轨迹。我们的ObjTac数据集可在以下网址找到：this https URL
英文摘要：
Recent vision-language-action (VLA) models build upon vision-language foundations, and have achieved promising results and exhibit the possibility of task generalization in robot manipulation. However, due to the heterogeneity of tactile sensors and the difficulty of acquiring tactile data, current VLA models significantly overlook the importance of tactile perception and fail in contact-rich tasks. To address this issue, this paper proposes OmniVTLA, a novel architecture involving tactile sensing. Specifically, our contributions are threefold. First, our OmniVTLA features a dual-path tactile encoder framework. This framework enhances tactile perception across diverse vision-based and force-based tactile sensors by using a pretrained vision transformer (ViT) and a semantically-aligned tactile ViT (SA-ViT). Second, we introduce ObjTac, a comprehensive force-based tactile dataset capturing textual, visual, and tactile information for 56 objects across 10 categories. With 135K tri-modal samples, ObjTac supplements existing visuo-tactile datasets. Third, leveraging this dataset, we train a semantically-aligned tactile encoder to learn a unified tactile representation, serving as a better initialization for OmniVTLA. Real-world experiments demonstrate substantial improvements over state-of-the-art VLA baselines, achieving 96.9% success rates with grippers, (21.9% higher over baseline) and 100% success rates with dexterous hands (6.2% higher over baseline) in pick-and-place tasks. Besides, OmniVTLA significantly reduces task completion time and generates smoother trajectories through tactile sensing compared to existing VLA. Our ObjTac dataset can be found at https://readerek.github.io/Objtac.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
