机器人中的视频生成模型——应用、研究挑战与未来方向
Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directions
AI总结：
本文综述了视频生成模型在机器人学中的应用、研究挑战及未来方向，探讨了其在物理模拟、动作预测和策略评估中的作用及面临的挑战。
摘要：
视频生成模型已经作为高保真的物理世界模型出现，能够合成高质量的视频，捕捉代理与其环境之间细致互动的视频，这取决于多模态用户输入。它们令人印象深刻的能力解决了许多长期面临的物理模拟挑战，推动了在许多问题领域中的广泛应用，例如机器人学。例如，视频模型能够实现逼真的、物理一致的可变形体模拟，而无需做出具有抑制性的简化假设，这在物理模拟中是一个主要瓶颈。此外，视频模型可以作为基础世界模型，以细致和表达性的方式捕捉世界动态。因此，它们克服了仅使用语言抽象描述复杂物理互动时的有限表达性。在本文综述中，我们回顾了视频模型及其在机器人学中的应用作为具身世界模型，包括在模仿学习中的低成本数据生成和动作预测、强化学习中的动态和奖励建模、视觉规划以及策略评估。此外，我们强调了阻碍视频模型在机器人学中可信整合的重要挑战，包括指令跟随差、如违反物理规律的幻觉以及不安全的内容生成，以及根本限制，如显著的数据整理、训练和推理成本。我们提出了潜在的未来方向以解决这些开放性研究挑战，以激励研究并最终促进更广泛的应用，尤其是在安全关键领域。
英文摘要：
Video world models have emerged as promising candidates for high-fidelity world models, offering the potential to synthesize high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models can generate photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. However, despite their potential, video models still generate physics-violating future predictions, often manifesting as hallucinations. In this survey, we provide a review of video models and their applications as embodied world models in robotics, including efficient data generation and policy learning, dynamics and rewards modeling in reinforcement learning, policy evaluation, and visual planning. Further, we highlight important challenges hindering the trustworthy integration of video models, such as poor instruction following, hallucinations like violations of physics, unsafe content generation, in addition to significant data and compute overhead. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings. We provide a curated bibliography at https://github.com/irom-princeton/awesome-robotics-video-world-model-papers .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
