arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.13423cs.CV
收藏

解耦视频生成:基于免训练扩散模型专家链的框架

Decoupled Video Generation with Chain of Training-free Diffusion Model Experts

University of Sydney(悉尼大学);Southeast University(东南大学);Central South University(中南大学);Shanghai Jiaotong University(上海交通大学);Sensetime Research(商汤科技研究院);Hong Kong University of Science and Technology(香港科技大学)

Wenhao Li, Yichao Cao, Xiu Su, Xi Lin, Shan You, Mingkai Zheng, Yi Chen, Chang Xu

AI总结:

本文提出 ConFiner 框架,将视频生成解耦为结构控制与时空细化,通过免训练扩散模型专家链及协调去噪实现高效生成,仅用 10% 推理成本即超越代表性模型,并扩展 ConFiner-Long 支持 600 帧长视频生成。

摘要:

视频生成模型在电影制作等领域具有巨大潜力。然而,由于视频生成任务的极高复杂性,当前的视频扩散模型需要高昂的计算成本,且生成结果不尽如人意。本文提出了 ConFiner,一个高效的视频生成框架,将视频生成解耦为更简单的子任务:结构控制与时空细化。该框架可通过现成的扩散模型专家链生成高质量视频,每个专家负责一个解耦的子任务。在细化过程中,我们引入了协调去噪方法,能够将多个扩散专家的能力融合到单次采样中。此外,我们设计了 ConFiner-Long 框架,通过三种约束策略在 ConFiner 上生成长时连贯视频。实验结果表明,仅使用 10% 的推理成本,我们的 ConFiner 在所有客观和主观指标上均超越了 Lavie 和 Modelscope 等代表性模型。ConFiner-Long 能够生成高达 600 帧的高质量连贯视频。

英文摘要:

Video generation models hold substantial potential in areas such as filmmaking. However, current video diffusion models need high computational costs and produce suboptimal results due to extreme complexity of video generation task. In this paper, we propose \textbf{ConFiner}, an efficient video generation framework that decouples video generation into easier subtasks: structure \textbf{con}trol and spatial-temporal re\textbf{fine}ment. It can generate high-quality videos with chain of off-the-shelf diffusion model experts, each expert responsible for a decoupled subtask. During the refinement, we introduce coordinated denoising, which can merge multiple diffusion experts' capabilities into a single sampling. Furthermore, we design ConFiner-Long framework, which can generate long coherent video with three constraint strategies on ConFiner. Experimental results indicate that with only 10\% of the inference cost, our ConFiner surpasses representative models like Lavie and Modelscope across all objective and subjective metrics. And ConFiner-Long can generate high-quality and coherent videos with up to 600 frames.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑