arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.10076cs.AIcs.LG
收藏

VideoAgent:自我改进的视频生成

VideoAgent: Self-Improving Video Generation

University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Stanford University(斯坦福大学);Google Deepmind(谷歌DeepMind);New York University(纽约大学)

Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, Sherry Yang

AI总结:

针对视频生成用于机器人控制时存在幻觉内容和物理不真实的问题,提出VideoAgent,通过自我条件一致性优化视频规划并利用环境反馈自我改进,在模拟实验中大幅提升任务成功率,为视频生成锚定物理世界提供有效工具。

摘要:

视频生成已被用于为机器人系统控制生成视觉规划。给定图像观测和语言指令,先前的工作生成视频规划,然后将其转换为机器人控制指令执行。然而,利用视频生成进行控制的主要瓶颈在于生成视频的质量,这些视频通常存在幻觉内容和不真实的物理现象,导致从生成视频中提取控制动作时任务成功率较低。虽然扩大数据集和模型规模提供了部分解决方案,但整合外部反馈对于将视频生成锚定在现实世界中既自然又必要。基于这一观察,我们提出VideoAgent,用于基于外部反馈自我改进生成的视频规划。VideoAgent不直接执行生成的视频规划,而是首先使用一种称为自我条件一致性的新过程优化生成的视频规划,使推理时计算能够转化为更好的生成视频规划。在优化后的视频规划执行过程中,VideoAgent可以从环境中收集额外数据,进一步改进视频规划生成。在MetaWorld和iTHOR的模拟机器人操作实验中,VideoAgent大幅减少了幻觉,从而提高了下游操作任务的成功率。我们进一步说明,VideoAgent可以有效优化真实机器人视频,为机器人可作为将视频生成锚定在物理世界的有效工具提供了早期指标。视频演示和代码可在https://video-as-agent.github.io获取。

英文摘要:

Video generation has been used to generate visual plans for controlling robotic systems. Given an image observation and a language instruction, previous work has generated video plans which are then converted to robot controls to be executed. However, a major bottleneck in leveraging video generation for control lies in the quality of the generated videos, which often suffer from hallucinatory content and unrealistic physics, resulting in low task success when control actions are extracted from the generated videos. While scaling up dataset and model size provides a partial solution, integrating external feedback is both natural and essential for grounding video generation in the real world. With this observation, we propose VideoAgent for self-improving generated video plans based on external feedback. Instead of directly executing the generated video plan, VideoAgent first refines the generated video plans using a novel procedure which we call self-conditioning consistency, allowing inference-time compute to be turned into better generated video plans. As the refined video plan is being executed, VideoAgent can collect additional data from the environment to further improve video plan generation. Experiments in simulated robotic manipulation from MetaWorld and iTHOR show that VideoAgent drastically reduces hallucination, thereby boosting success rate of downstream manipulation tasks. We further illustrate that VideoAgent can effectively refine real-robot videos, providing an early indicator that robots can be an effective tool in grounding video generation in the physical world. Video demos and code can be found at https://video-as-agent.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑