arXiv 2509.266422026-03-18cs.RO

MLA：一种多感官语言-动作模型，用于机器人操作中的多模态理解和预测

MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Beijing Innovation Center of Humanoid Robotics (X-Humanoid)（北京类人机器人创新中心（X-Humanoid））；The Chinese University of Hong Kong (CUHK)（香港中文大学（CUHK））

Zhuoyang Liu, Jiaming Liu, Jiadong Xu, Nuowei Han, Chenyang Gu, Hao Chen, Kaichen Zhou, Renrui Zhang, Kai Chin Hsieh, Kun Wu, Zhengping Che, Jian Tang, Shanghang Zhang

备注：: Project page: https://robotic-mla.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MLA模型，通过多感官融合与未来目标预测，提升机器人在复杂接触任务中的操控能力，实验显示其在2D和3D任务中性能优于现有方法。

摘要：

本文提出MLA模型，通过多感官融合与未来目标预测，提升机器人在复杂接触任务中的操控能力，实验显示其在2D和3D任务中性能优于现有方法。

英文摘要：

Vision-language-action models (VLAs) have shown generalization capabilities in robotic manipulation tasks by inheriting from vision-language models (VLMs) and learning action generation. Most VLA models focus on interpreting vision and language to generate actions, whereas robots must perceive and interact within the spatial-physical world. This gap highlights the need for a comprehensive understanding of robotic-specific multisensory information, which is crucial for achieving complex and contact-rich control. To this end, we introduce a multisensory language-action (MLA) model that collaboratively perceives heterogeneous sensory modalities and predicts future multisensory objectives to facilitate physical world modeling. Specifically, to enhance perceptual representations, we propose an encoder-free multimodal alignment scheme that innovatively repurposes the large language model itself as a perception module, directly interpreting multimodal cues by aligning 2D images, 3D point clouds, and tactile tokens through positional correspondence. To further enhance MLA's understanding of physical dynamics, we design a future multisensory generation post-training strategy that enables MLA to reason about semantic, geometric, and interaction information, providing more robust conditions for action generation. For evaluation, the MLA model outperforms the previous state-of-the-art 2D and 3D VLA methods by 12% and 24% in complex, contact-rich real-world tasks, respectively, while also demonstrating improved generalization to unseen configurations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。