DiffSynth：面向真实感视频合成的潜在迭代内去闪烁方法
DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis
AI总结：
本文提出 DiffSynth，通过潜在迭代内去闪烁框架和 patch blending 算法将图像扩散模型改造为视频合成流程，有效抑制闪烁并提升视频一致性，适用于多种视频合成任务。
摘要：
近年来，扩散模型已成为图像合成中最强大的方法。然而，将这些模型直接应用于视频合成面临挑战，因为通常会导致明显的闪烁内容。尽管最近提出的零样本方法在一定程度上可以缓解闪烁，但我们仍然难以生成连贯的视频。在本文中，我们提出了 DiffSynth，这是一种旨在将图像合成流程转换为视频合成流程的新方法。DiffSynth 由两个关键组件组成：潜在迭代内去闪烁框架和视频去闪烁算法。潜在迭代内去闪烁框架将视频去闪烁应用于扩散模型的潜在空间，有效防止中间步骤中的闪烁累积。此外，我们提出了一种名为 patch blending 算法的视频去闪烁算法，该算法重新映射不同帧中的对象并将其混合在一起，以增强视频一致性。DiffSynth 的显著优势之一是其对各种视频合成任务的广泛适用性，包括文本引导的视频风格化、时尚视频合成、图像引导的视频风格化、视频修复和 3D 渲染。在文本引导的视频风格化任务中，我们使得无需精挑细选即可合成高质量视频成为可能。实验结果证明了 DiffSynth 的有效性。所有视频均可在我们的项目页面上查看。源代码也将发布。
英文摘要：
In recent years, diffusion models have emerged as the most powerful approach in image synthesis. However, applying these models directly to video synthesis presents challenges, as it often leads to noticeable flickering contents. Although recently proposed zero-shot methods can alleviate flicker to some extent, we still struggle to generate coherent videos. In this paper, we propose DiffSynth, a novel approach that aims to convert image synthesis pipelines to video synthesis pipelines. DiffSynth consists of two key components: a latent in-iteration deflickering framework and a video deflickering algorithm. The latent in-iteration deflickering framework applies video deflickering to the latent space of diffusion models, effectively preventing flicker accumulation in intermediate steps. Additionally, we propose a video deflickering algorithm, named patch blending algorithm, that remaps objects in different frames and blends them together to enhance video consistency. One of the notable advantages of DiffSynth is its general applicability to various video synthesis tasks, including text-guided video stylization, fashion video synthesis, image-guided video stylization, video restoring, and 3D rendering. In the task of text-guided video stylization, we make it possible to synthesize high-quality videos without cherry-picking. The experimental results demonstrate the effectiveness of DiffSynth. All videos can be viewed on our project page. Source codes will also be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
