arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.02617cs.LGcs.AIcs.CV
收藏

通过AI反馈提升文本到视频生成中动态物体交互

Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback

Google DeepMind(谷歌DeepMind);The University of Tokyo(东京大学);Stanford University(斯坦福大学)

Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, Sherry Yang

AI总结:

本文研究如何利用反馈提升文本到视频模型中动态物体交互的质量,提出利用视觉语言模型提供针对性反馈,实验表明该方法在视频交互场景质量上有显著提升。

摘要:

大型文本到视频模型在多种下游应用中具有巨大潜力,但难以准确描绘动态物体交互,常导致不真实的运动和违反现实物理的错误。受大型语言模型启发,我们探索利用外部反馈来提高文本到视频模型的质量。我们旨在回答关键问题:哪些类型的反馈与特定的自我改进算法结合,最能有效克服运动不一致和现实物体交互?我们指出,文本到视频模型的离线强化学习微调算法可以等效于统一概率目标。这一视角强调,原则上没有算法主导方法,而应关注奖励和数据的属性。尽管人类反馈可扩展性差,但视觉语言模型能像人类一样感知视频场景。我们提出利用视觉语言模型提供专门针对视频中物体动态的感知反馈。与流行的视频质量度量标准相比,实验表明,我们的方法结合二进制AI反馈在视频交互场景质量上取得最大提升,经AI、人类和质量度量评估验证。值得注意的是,使用视觉语言模型信号时,在多个物体复杂交互和物体下落真实描绘场景中观察到显著提升。

英文摘要:

Large text-to-video models hold immense potential for a wide range of downstream applications. However, they struggle to accurately depict dynamic object interactions, often resulting in unrealistic movements and frequent violations of real-world physics. One solution inspired by large language models is to align generated outputs with desired outcomes using external feedback. In this work, we investigate the use of feedback to enhance the quality of object dynamics in text-to-video models. We aim to answer a critical question: what types of feedback, paired with which specific self-improvement algorithms, can most effectively overcome movement misalignment and realistic object interactions? We first point out that offline RL-finetuning algorithms for text-to-video models can be equivalent as derived from a unified probabilistic objective. This perspective highlights that there is no algorithmically dominant method in principle; rather, we should care about the property of reward and data. While human feedback is less scalable, vision-language models could notice the video scenes as humans do. We then propose leveraging vision-language models to provide perceptual feedback specifically tailored to object dynamics in videos. Compared to popular video quality metrics measuring alignment or dynamics, the experiments demonstrate that our approach with binary AI feedback drives the most significant improvements in the quality of interaction scenes in video, as confirmed by AI, human, and quality metric evaluations. Notably, we observe substantial gains when using signals from vision language models, particularly in scenarios involving complex interactions between multiple objects and realistic depictions of objects falling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑