在统一因果训练下,世界建模与动作建模能否相互促进?
Can World and Action Modeling Benefit Each Other under Unified Causal Training?
AI总结:
该研究提出OneWM框架,通过统一因果训练探究世界建模与动作建模的相互促进作用,经LIBERO等具身测试平台验证,二者可实现双向性能提升。
摘要:
世界建模与动作建模密切相关,但二者在统一因果训练下的相互益处仍不明确。我们提出OneWM,这是一个用于统一因果训练的简单框架,在共享的视频-动作架构中将动作同时视为世界预测的条件和策略学习的目标。利用该框架,我们研究基于指定动作学习正向动力学是否能改进动作学习,以及学习生成动作是否反过来能改进基于动作的世界建模。我们使用LIBERO、LIBERO-Plus和RoboCasa365作为具身测试平台,评估策略学习和世界建模,并辅以真实世界的策略实验。学习指定动作的视觉后果可提升策略性能和场景变化下的泛化能力;反过来,动作学习可提升同一架构下多视图世界预测的保真度。这些结果对两个问题均给出肯定答案,支持在统一因果训练下世界建模与动作生成之间存在相互益处。
英文摘要:
World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。