基于级联条件扩散的文本驱动说话视频编辑
Text-based Talking Video Editing with Cascaded Conditional Diffusion
AI总结：
提出级联条件扩散框架，分音频到稠密地标运动和运动到视频两阶段，实现高效、身份保持的文本驱动说话视频编辑。
摘要：
基于文本的说话人头视频编辑旨在通过用户友好的文本编辑方式，高效地插入、删除和替换说话视频的片段。这一任务具有挑战性，原因在于：1）可泛化的说话人脸表示，2）无缝的视听过渡，以及3）保持身份一致的说话人脸。以往的工作要么需要数分钟的说话人脸视频训练数据以及昂贵的测试时优化来实现定制化的说话视频编辑，要么直接生成视频序列而不考虑上下文信息，导致泛化表示差、过渡不连贯，甚至身份不一致。本文提出了一种高效的级联条件扩散框架，包含两个阶段：音频到稠密地标运动，以及运动到视频。在第一阶段，我们首先提出一个动态加权上下文扩散模块，根据编辑后的音频合成稠密地标运动。在第二阶段，我们引入一个扭曲引导的条件扩散模块。该模块首先在编辑区间的起始帧和结束帧之间进行插值，以生成平滑的中间帧。然后，借助音频到稠密运动图像，这些中间帧被扭曲以获得粗糙的中间帧。以扭曲后的中间帧为条件，采用扩散模型生成细节丰富的高分辨率目标帧，从而保证连贯且保持身份的过渡。级联条件扩散模型将复杂的说话编辑任务分解为两个灵活的子生成任务，在小数据集上提供了可泛化的说话人脸表示、无缝的视听过渡和保持身份一致的人脸。实验表明，所提方法有效且优越。
英文摘要：
Text-based talking-head video editing aims to efficiently insert, delete, and substitute segments of talking videos through a user-friendly text editing approach. It is challenging because of \textbf{1)} generalizable talking-face representation, \textbf{2)} seamless audio-visual transitions, and \textbf{3)} identity-preserved talking faces. Previous works either require minutes of talking-face video training data and expensive test-time optimization for customized talking video editing or directly generate a video sequence without considering in-context information, leading to a poor generalizable representation, or incoherent transitions, or even inconsistent identity. In this paper, we propose an efficient cascaded conditional diffusion-based framework, which consists of two stages: audio to dense-landmark motion and motion to video. \textit{\textbf{In the first stage}}, we first propose a dynamic weighted in-context diffusion module to synthesize dense-landmark motions given an edited audio. \textit{\textbf{In the second stage}}, we introduce a warping-guided conditional diffusion module. The module first interpolates between the start and end frames of the editing interval to generate smooth intermediate frames. Then, with the help of the audio-to-dense motion images, these intermediate frames are warped to obtain coarse intermediate frames. Conditioned on the warped intermedia frames, a diffusion model is adopted to generate detailed and high-resolution target frames, which guarantees coherent and identity-preserved transitions. The cascaded conditional diffusion model decomposes the complex talking editing task into two flexible generation tasks, which provides a generalizable talking-face representation, seamless audio-visual transitions, and identity-preserved faces on a small dataset. Experiments show the effectiveness and superiority of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
