Agile-WAM：用于接触丰富机器人控制的敏捷触觉世界动作模型

中文摘要 AI 辅助

世界动作模型（WAMs）超越了传统的视觉运动策略，通过联合预测未来的世界状态和机器人动作，使策略能够学习支持有效控制的物理动力学。然而，最近的触觉WAMs通常依赖大规模预训练生成骨干网络来捕捉接触丰富的物理动力学，这限制了其推理效率和灵活部署。在本文中，我们提出了\ABBR{}，一种用于接触丰富机器人控制的敏捷触觉世界动作模型。\ABBR{}将视觉和触觉观测编码到一个共享潜空间中，作为直接的视觉-触觉到动作流匹配过程的源，该过程可以联合生成动作块和未来视觉/触觉潜变量的潜在表示。一个关键观察是，视觉和触觉信号在本质上以不同的时间尺度演化：相邻的视觉帧通常高度相似，而触觉信号在接触时可能突然变化。因此，我们在\ABBR{}中引入了多视界多模态预测，它在较大的时间偏移处为视觉潜变量提供监督，同时预测下一帧的触觉潜变量以捕捉细粒度的接触动力学。在九个模拟和五个真实世界的接触丰富操作任务中，\ABBR{}展示了强大且稳健的性能，在成功率上优于最强基线，同时保持低推理延迟。特别是，在五个真实世界实验中，\ABBR{}在总体成功率上取得了$\textbf{29.4\\%}$的相对提升，同时实现了$\textbf{11.9 ms}$的推理延迟。这些结果表明，多模态WAM可以通过适合精确和高频机器人控制的敏捷架构来实现。更多细节可在我们的项目页面获取：此https URL。

英文摘要

World Action Models (WAMs) advance beyond conventional visuomotor policies by jointly predicting future world states and robot actions, enabling the policy to learn phys- ical dynamics that support effective control. However, recent tactile WAMs often rely on large-scale pretrained generative backbones to capture contact-rich physical dynamics, which limit their inference efficiency and flexible deployment. In this paper, we present Agile-WAM, an agile tactile World Action Model for contact-rich robot control. Agile-WAM encodes visual and tactile observations into a shared latent that serves as the source of a direct vision-tactile-to-action flow-matching process, which can jointly generate latent representations of action chunks and future visual/tactile latents. A key observation is that vision and tactile signals evolve at inherently different timescales: adjacent visual frames are often highly similar, whereas tactile signals can change abruptly upon contact. We therefore introduce multi-horizon multimodal prediction in Agile-WAM, which provides supervision for visual latent at a larger temporal offset while predicting the tactile latent in the next frame to capture fine-grained contact dynamics. Across nine simulated and five real-world contact-rich ma- nipulation tasks, Agile-WAM demonstrates strong and robust performance, outperforming the strongest baseline in success rate while maintaining low inference latency. In particular, in five real-world experiments, Agile-WAM yields a relative gain of 29.4% in overall success rates while achieving inference latency of 11.9 ms. These results demonstrate that multimodal WAM can be achieved with an agile architecture suitable for precise and high-frequency robot control. More details are available on our project page.

发表机构

• University of California, Davis（加州大学戴维斯分校）
• Analog Devices（亚德诺半导体）

机构由 AI 辅助整理，请以论文原文为准。