arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.05082cs.CV
收藏

CoNo:面向免调优长视频扩散的一致性噪声注入

CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion

University of Science and Technology of China(中国科学技术大学)

Xingrui Wang, Xin Li, Zhibo Chen

AI总结:

针对免调优长视频扩散中的场景一致性问题,本文提出 CoNo,通过“回看”机制和长时一致性正则化增强视频片段间细粒度过渡并抑制内容偏移,在单文本和多文本条件下均验证了有效性。

摘要:

免调优长视频扩散方法已被提出,通过复用预训练短视频扩散模型的知识而无需重新训练,来生成内容更丰富的长时视频。然而,大多数工作忽视了细粒度的长时视频一致性建模,导致场景一致性有限(例如不合理的物体或背景切换),尤其是在多个文本输入的情况下。为缓解这一问题,我们提出了一致性噪声注入方法,简称 CoNo,该方法引入“回看”机制以增强不同视频片段之间的细粒度场景过渡,并设计了长时一致性正则化,以在通过噪声预测扩展视频内容时消除内容偏移。具体而言,“回看”机制将噪声调度过程拆分为三个关键部分,其中一个内部噪声预测部分被注入到两个视频扩展部分中,旨在实现两个视频片段之间的细粒度过渡。长时一致性正则化专注于显式最小化扩展视频片段与原始视频片段预测噪声之间的逐像素距离,从而防止场景突变。大量实验表明,上述策略在单文本和多文本提示条件下进行长视频生成时均具有有效性。项目已发布于 https://wxrui182.github.io/CoNo.github.io/。

英文摘要:

Tuning-free long video diffusion has been proposed to generate extended-duration videos with enriched content by reusing the knowledge from pre-trained short video diffusion model without retraining. However, most works overlook the fine-grained long-term video consistency modeling, resulting in limited scene consistency (i.e., unreasonable object or background transitions), especially with multiple text inputs. To mitigate this, we propose the Consistency Noise Injection, dubbed CoNo, which introduces the "look-back" mechanism to enhance the fine-grained scene transition between different video clips, and designs the long-term consistency regularization to eliminate the content shifts when extending video contents through noise prediction. In particular, the "look-back" mechanism breaks the noise scheduling process into three essential parts, where one internal noise prediction part is injected into two video-extending parts, intending to achieve a fine-grained transition between two video clips. The long-term consistency regularization focuses on explicitly minimizing the pixel-wise distance between the predicted noises of the extended video clip and the original one, thereby preventing abrupt scene transitions. Extensive experiments have shown the effectiveness of the above strategies by performing long-video generation under both single- and multi-text prompt conditions. The project has been available in https://wxrui182.github.io/CoNo.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑