发表机构
Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
DeCAL提出了一种物理基础的灵巧视觉-语言-动作模型，通过接触感知门控融合和潜在共同想象整合触觉信息，在灵巧操作任务中达到71%平均成功率，并展现出强泛化能力。
AI 中文摘要
灵巧操作涉及与物理世界的丰富接触和细粒度交互，由于严重的视觉遮挡和复杂的接触动力学，给现有的视觉-语言-动作（VLA）模型带来了重大挑战。尽管最近的工作已将触觉感知纳入机器人操作，但大多数方法仍依赖于同质多模态融合，缺乏自适应的触觉整合和物理动力学的显式建模。在这项工作中，我们提出了DeCAL，一种物理基础的灵巧视觉-语言-动作模型，它统一了接触丰富灵巧操作的理解、想象和动作生成。基于混合变换器（MoT）架构，DeCAL利用专门的专家处理每种能力，同时实现它们之间的高效信息流动。为了有效利用触觉信息，我们引入了自适应视觉-触觉融合，通过接触感知的门控策略动态调节触觉交互。此外，我们提出了视觉-触觉潜在共同想象，以联合建模视觉和触觉动力学，使策略具备隐式的物理世界知识。实验结果表明，DeCAL在所有任务中持续实现了最先进的性能，达到了71%的平均成功率和83.4%的进度成功率，同时展示了对未见场景的强大泛化能力。网站可在此https URL获取。
英文摘要
Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios. The website is available at https://aureleopku.github.io/DeCAL.
