模块化感觉流：整合物理反馈在视觉-语言-动作模型中
Modular Sensory Stream for Integrating Physical Feedback in Vision-Language-Action Models
AI总结：
本文提出MoSS框架，通过模块化感觉流整合多种物理信号，提升视觉-语言-动作模型的行动预测能力。
摘要：
人类通过超越视觉感知的多种物理反馈来理解和互动现实世界。受此启发，近期方法尝试将物理感官信号整合到视觉-语言-动作模型（VLAs）中。然而，它们通常专注于单一类型的物理信号，未能捕捉到现实世界交互的异质性和互补性。本文提出MoSS，一种模块化感觉流框架，使VLAs能够利用多种感官信号进行动作预测。具体而言，我们引入了解耦的模态流，通过联合跨模态自注意力将异构物理信号整合到动作流中。为了使新模态的稳定整合，我们采用两阶段训练方案，在早期阶段冻结预训练的VLA参数。此外，为了更好地捕捉接触交互动态，我们引入了预测未来物理信号的辅助任务。通过广泛的现实世界实验，我们证明MoSS成功增强了VLAs，以利用多种物理信号（即触觉和扭矩），整合多种信号以实现协同性能提升。
英文摘要：
Humans understand and interact with the real world by relying on diverse physical feedback beyond visual perception. Motivated by this, recent approaches attempt to incorporate physical sensory signals into Vision-Language-Action models (VLAs). However, they typically focus on a single type of physical signal, failing to capture the heterogeneous and complementary nature of real-world interactions. In this paper, we propose MoSS, a modular sensory stream framework that adapts VLAs to leverage multiple sensory signals for action prediction. Specifically, we introduce decoupled modality streams that integrate heterogeneous physical signals into the action stream via joint cross-modal self-attention. To enable stable incorporation of new modalities, we adopt a two-stage training scheme that freezes pretrained VLA parameters in the early stage. Furthermore, to better capture contact interaction dynamics, we incorporate an auxiliary task that predicts future physical signals. Through extensive real-world experiments, we demonstrate that MoSS successfully augments VLAs to leverage diverse physical signals (i.e., tactile, force, and torque), integrating multiple signals to achieve synergistic performance gains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
