arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.09293cs.RO
收藏

MDF++:具有端到端表示学习的视觉-触觉世界动作模型

MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning

University of Michigan, Ann Arbor(密歇根大学安娜堡分校)

Zixuan Huang, Jinhao Yu, Zachary Charlick, Dmitry Berenson

AI总结:

本文提出MDF++视觉-触觉世界动作模型,以扩散网络为骨干端到端学习多感官观测表示,通过多任务学习避免特征坍塌,在接触密集操作任务上性能较基准提升最高50%。

摘要:

近年来,世界动作模型(WAMs)已成为机器人操作领域的强大框架,通过联合建模动作生成与未来观测预测,WAMs可执行动作并预判动作结果。然而,现有WAMs通常依赖经重建训练的冻结预训练编码器,其生成的表示可能不适用于机器人控制。本文针对WAMs学习基于物理、与控制相关的表示这一问题,提出MDF++——一种视觉-触觉世界动作模型,该模型以扩散网络为骨干,端到端学习多感官观测的潜在表示。尽管直接联合训练可能导致特征坍塌,MDF++引入多任务学习策略,利用跨模态相关性学习任务相关表示。我们在一系列接触密集的操作任务(包括真实世界的拧螺栓和螺母装配)上评估MDF++,结果显示其性能较基于冻结预训练表示的WAMs提升最高达50%。更多视频可在我们的网站查看。

英文摘要:

Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruction, which may not produce representations well suited for robot control. In this paper, we tackle the problem of learning physicallygrounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑