PhysRAG: 通过检索增强生成提升视频生成中的物理感知
PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
AI总结：
提出PhysRAG管道，利用检索增强生成（RAG）和可学习查询注入物理知识，在7K高质量视频上训练，在PhyGenBench和VBench上达到最优视觉质量和物理规则遵从性。
摘要：
由于难以捕捉热力学、力学和光学等多样物理现象，开发具有物理感知的视频生成模型仍然是一个重大挑战。在这项工作中，我们引入了PhysRAG，一种通过检索增强生成（RAG）增强视频生成中物理感知的新型管道。为了解决高质量数据有限的问题，我们基于WISA-80K数据集设计了一个两阶段数据过滤管道，从而得到一组精选的7K高质量视频用于训练。此外，我们构建了一个物理视频数据库，并开发了一种使用可学习查询将物理知识注入视频扩散模型的机制。我们的方法在视觉质量和物理规则遵从性方面均达到了最先进的性能，在PhyGenBench和VBench等基准测试中超越了现有模型。我们进行了广泛的消融研究，以验证我们关键组件的有效性，包括数据过滤管道、RAG机制和物理信息提取方法。为促进未来研究，我们的代码、数据和模型准备在此https URL发布。
英文摘要：
Developing physically aware video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics. In this work, we introduce PhysRAG, a novel pipeline that enhances physical awareness in video generation through Retrieval-Augmented Generation (RAG). To address the issue of limited high-quality data, we design a two-stage data filtering pipeline based on the WISA-80K dataset, resulting in a curated set of 7K high-quality videos for training. Furthermore, we construct a physical video database and develop a mechanism to inject physical knowledge into a video diffusion model using learnable queries. Our method achieves state-of-the-art performance in both visual quality and physical rule compliance, surpassing existing models in benchmarks such as PhyGenBench and VBench. We conduct extensive ablation studies to validate the effectiveness of our key components, including the data filtering pipeline, RAG mechanism, and method for physical information extraction. To facilitate future research, our code, data, and models are prepared for release at https://github.com/sediment1024/PhysRAG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
