arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.16641cs.RO
收藏

什么限制了世界动作模型的跨任务泛化能力?

What Limits Cross-Task Generalization in World Action Models?

Renmin University of China(中国人民大学);Meituan(美团)

Chen Zhao, Zipeng Ye, Yihan Lin, Yuchen Xie, Jing Zhang

AI总结:

该研究针对世界动作模型跨任务泛化差的问题,通过解耦模型引入AoE接口诊断出失效区域,提出VITA损失将生成导向动作相关细节,使执行成功率提升至47.0%并可迁移。

摘要:

世界动作模型(WAMs)的跨任务泛化仍然困难,即便其预测的未来场景呈现出合理的操作效果。我们通过将WAM解耦为世界模型和逆动力学模型(IDM),同时固定IDM并替换未来图像的来源,使这一差距可直接测试。我们引入了“一次对齐,处处评估(AoE)”,这是一个在大规模机器人数据上预训练的相机空间逆动力学模型,为不同任务和环境的未来来源提供了恢复接口。通过该接口,我们发现从未见任务的记录未来中恢复的动作执行效果几乎与重放的演示相当,而世界模型预测的未来在各个基准测试中均失效。这些失效并非普遍存在,而是集中在夹爪和交互几何区域,生成能力则被用于任务无关的背景。这一诊断表明,监督应精准针对这些区域。因此,我们引入了视觉到动作(VITA)损失,该损失将生成的未来通过冻结的AoE,并惩罚恢复动作的差异,从而将生成导向与动作相关的细节。在两个世界模型和三个操作基准测试中,这一简单改动将生成未来的执行成功率从接近零提升至47.0%。相同的损失也可迁移至WAMs,并在原生控制中取得一致的增益。

英文摘要:

Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑