从预训练视频模型中学习物理：一种多模态连续和序列世界交互模型用于机器人操作
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
AI总结：
PhysGen通过预训练视频模型学习物理知识，实现机器人操作的连续和序列世界交互，优于现有基线方法。
摘要：
大规模机器人数据的稀缺性推动了从其他模态中重新利用基础模型用于策略学习。在本工作中，我们引入了PhysGen（从预训练视频生成模型中学习物理），一种可扩展的连续和序列世界交互框架，利用自回归视频生成来解决机器人操作任务。通过将预训练视频模型视为物理模拟器的代理，PhysGen建模了外部环境与机器人动作之间的动态相互作用。我们引入了一种多模态连续表示，将视频和动作统一到共享的物理令牌中，弥合了离散视频生成与连续机器人控制之间的差距。这种方法使隐含的物理知识——如物体的持续性和动力学——从视频预训练无缝转移到下游操作中。为了确保高效的收敛，我们结合了因果掩码、逆运动学、Lookahead Multi-Token Prediction（L-MTP）和键值（KV）缓存。在Libero和ManiSkill基准测试中，PhysGen的结果表明，它在稳健的基线中表现一致，超过了OpenVLA和WorldVLA，分别以13.8%和8.8%的幅度。值得注意的是，在现实世界场景中，PhysGen的表现与大规模动作预训练模型如π₀相当，无需先前的动作特定预训练，这在如抓取透明物体等复杂的物理任务中表现出更强的能力。这些发现验证了从预训练视频生成器中提取物理直觉以促进通用机器人操作的潜力。
英文摘要：
The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work, we introduce PhysGen (Learning Physics from Pretrained Video Generation Models), a scalable continuous and sequential world interaction framework that leverages autoregressive video generation to solve robotic manipulation tasks. By treating the pretrained video model as a proxy for a physics simulator, PhysGen models the dynamic interplay between the external environment and robot actions. We introduce a multimodal continuous representation that unifies video and action into shared physical tokens, bridging the gap between discrete video generation and continuous robotic control. This approach enables the seamless transfer of implicit physical knowledge-such as object permanence and dynamics-from video pretraining to downstream manipulation.To ensure efficient convergence, we incorporate causal masking, inverse kinematics, Lookahead Multi-Token Prediction (L-MTP), and key-value (KV) caching. Experimental results on the Libero and ManiSkill benchmarks demonstrate that PhysGen consistently outperforms robust baselines, surpassing OpenVLA and WorldVLA by margins of 13.8% and 8.8%, respectively. Notably, in real-world scenarios, PhysGen matches the performance of large-scale action-pretrained models like $π_0$ without requiring prior action-specific pretraining, demonstrating superior capability in physically complex tasks such as grasping transparent objects. These findings validate the potential of extracting physical intuition from pretrained video generators to facilitate generalizable robotic manipulation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
