DAE-Talker：基于扩散自编码器的高保真语音驱动说话人脸生成
DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
AI总结：
提出DAE-Talker，利用扩散自编码器提取数据驱动的潜在表示，结合Conformer的speech2latent模型，实现高保真、唇形同步且姿势可控的语音驱动说话人脸生成。
摘要：
尽管最近的研究在语音驱动说话人脸生成方面取得了显著进展，但生成视频的质量仍落后于真实录像。其中一个原因是使用了手工制作的中间表示（如面部landmarks和3DMM系数），这些表示基于人类知识设计，不足以精确描述面部运动。此外，这些方法需要外部预训练模型来提取这些表示，其性能为说话人脸生成设定了上限。为了解决这些限制，我们提出了一种名为DAE-Talker的新方法，该方法利用从扩散自编码器（DAE）获取的数据驱动的潜在表示。DAE包含一个将图像编码为潜在向量的图像编码器，以及一个从中重建图像的DDIM图像解码器。我们在说话人脸视频帧上训练DAE，然后提取其潜在表示作为基于Conformer的speech2latent模型的训练目标。这使得DAE-Talker能够合成完整视频帧，并产生与语音内容对齐的自然头部运动，而不是依赖模板视频中预先确定的头部姿势。我们还在speech2latent中引入了姿势建模以实现姿势可控性。此外，我们提出了一种使用在单个帧上训练的DDIM图像解码器生成连续视频帧的新方法，消除了直接对连续帧的联合分布进行建模的需要。我们的实验表明，DAE-Talker在唇形同步、视频保真度和姿势自然度方面优于现有的流行方法。我们还进行了消融研究以分析所提出技术的有效性，并展示了DAE-Talker的姿势可控性。
英文摘要：
While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reason for this is the use of handcrafted intermediate representations like facial landmarks and 3DMM coefficients, which are designed based on human knowledge and are insufficient to precisely describe facial movements. Additionally, these methods require an external pretrained model for extracting these representations, whose performance sets an upper bound on talking face generation. To address these limitations, we propose a novel method called DAE-Talker that leverages data-driven latent representations obtained from a diffusion autoencoder (DAE). DAE contains an image encoder that encodes an image into a latent vector and a DDIM image decoder that reconstructs the image from it. We train our DAE on talking face video frames and then extract their latent representations as the training target for a Conformer-based speech2latent model. This allows DAE-Talker to synthesize full video frames and produce natural head movements that align with the content of speech, rather than relying on a predetermined head pose from a template video. We also introduce pose modelling in speech2latent for pose controllability. Additionally, we propose a novel method for generating continuous video frames with the DDIM image decoder trained on individual frames, eliminating the need for modelling the joint distribution of consecutive frames directly. Our experiments show that DAE-Talker outperforms existing popular methods in lip-sync, video fidelity, and pose naturalness. We also conduct ablation studies to analyze the effectiveness of the proposed techniques and demonstrate the pose controllability of DAE-Talker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
