ConViTac：利用对比表示对齐视觉-触觉融合
ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations
AI总结：
提出ConViTac网络，通过自监督对比学习预训练的CEC机制将视觉和触觉输入投影到统一潜在嵌入，并利用跨模态注意力耦合特征融合，在材料分类和抓取预测任务中准确率最高提升12.0%。
摘要：
视觉和触觉是机器人的两种基本感觉模态，提供互补信息以增强感知和操作任务。先前的研究尝试联合学习视觉-触觉表示以提取更多有意义的信息。然而，这些方法通常依赖直接组合（如特征相加和拼接）进行模态融合，这往往导致特征整合不佳。在本文中，我们提出ConViTac，这是一种视觉-触觉表示学习网络，旨在使用对比表示增强融合过程中的特征对齐。我们的关键贡献是对比嵌入条件化（CEC）机制，该机制利用通过自监督对比学习预训练的对比编码器，将视觉和触觉输入投影到统一的潜在嵌入中。这些嵌入通过跨模态注意力来耦合视觉-触觉特征融合，旨在对齐统一表示并增强下游任务的性能。我们进行了大量实验，证明ConViTac在现实世界中优于当前最先进的方法，以及我们提出的CEC机制的有效性，该机制在材料分类和抓取预测任务中将准确率提高了12.0%。
英文摘要：
Vision and touch are two fundamental sensory modalities for robots, offering complementary information that enhances perception and manipulation tasks. Previous research has attempted to jointly learn visual-tactile representations to extract more meaningful information. However, these approaches often rely on direct combination, such as feature addition and concatenation, for modality fusion, which tend to result in poor feature integration. In this paper, we propose ConViTac, a visual-tactile representation learning network designed to enhance the alignment of features during fusion using contrastive representations. Our key contribution is a Contrastive Embedding Conditioning (CEC) mechanism that leverages a contrastive encoder pretrained through self-supervised contrastive learning to project visual and tactile inputs into unified latent embeddings. These embeddings are used to couple visual-tactile feature fusion through cross-modal attention, aiming at aligning the unified representations and enhancing performance on downstream tasks. We conduct extensive experiments to demonstrate the superiority of ConViTac in real world over current state-of-the-art methods and the effectiveness of our proposed CEC mechanism, which improves accuracy by up to 12.0% in material classification and grasping prediction tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
