arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.00596cs.CVcs.AI
收藏

PhyT2V:面向基于物理的文本到视频生成的LLM引导迭代自优化

PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation

University of Pittsburgh(匹兹堡大学)

Qiyao Xue, Xiangyu Yin, Boyuan Yang, Wei Gao

AI总结:

针对T2V模型缺乏物理常识且难以泛化到分布外领域的问题,提出数据无关技术PhyT2V,通过在提示中引入思维链和回退推理扩展模型能力,使物理规则遵循度提升2.3倍。

摘要:

基于Transformer的扩散模型最近使文本到视频(T2V)生成成为可能,但当前的T2V模型在遵循现实世界常识和物理规则方面能力不足,原因在于其对物理真实性的理解有限以及时间建模的缺陷。现有解决方案要么是数据驱动的,要么需要额外的模型输入,但无法泛化到分布外领域。在本文中,我们提出了PhyT2V,这是一种新的数据无关的T2V技术,通过在T2V提示中引入思维链和回退推理,将当前T2V模型的视频生成能力扩展到分布外领域。我们的实验表明,PhyT2V将现有T2V模型对现实世界物理规则的遵循度提高了2.3倍,并且与T2V提示增强器相比实现了35%的提升。源代码可在https://github.com/pittisl/PhyT2V获取。

英文摘要:

Text-to-video (T2V) generation has been recently enabled by transformer-based diffusion models, but current T2V models lack capabilities in adhering to the real-world common knowledge and physical rules, due to their limited understanding of physical realism and deficiency in temporal modeling. Existing solutions are either data-driven or require extra model inputs, but cannot be generalizable to out-of-distribution domains. In this paper, we present PhyT2V, a new data-independent T2V technique that expands the current T2V model's capability of video generation to out-of-distribution domains, by enabling chain-of-thought and step-back reasoning in T2V prompting. Our experiments show that PhyT2V improves existing T2V models' adherence to real-world physical rules by 2.3x, and achieves 35% improvement compared to T2V prompt enhancers. The source codes are available at: https://github.com/pittisl/PhyT2V.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑