基于统一扩散生成器的多模态驱动说话人脸生成
Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generator
AI总结：
提出统一扩散生成器TGDM，利用文本情感控制与纹理几何感知去噪，实现多模态驱动的高保真说话人脸生成及人脸交换。
摘要：
多模态驱动的说话人脸生成是指利用从驱动图像和视频中迁移的给定姿态、表情和注视，或从文本和音频中估计出的这些信息，来动画化一张肖像。然而，现有方法忽视了文本模态的潜力，其生成器主要遵循源导向的特征重排范式，并耦合了不稳定的GAN框架。在这项工作中，我们首先在文本提示中表示情感，这可以从CLIP中继承丰富的语义，从而实现灵活且泛化的情感控制。我们进一步将这些任务重组为目标导向的纹理迁移，并采用扩散模型。更具体地说，给定一个带纹理的人脸作为源，以及从期望的3DMM系数投影出的渲染人脸作为目标，我们提出的纹理-几何感知扩散模型将复杂的迁移问题分解为多条件去噪过程，其中基于纹理注意力的模块精确建模了源和目标条件中所包含的外观与几何线索之间的对应关系，并融入额外的隐式信息以实现高保真的说话人脸生成。此外，TGDM可以优雅地适用于人脸交换。我们推导出一种无需不稳定的跷跷板式优化的新范式，从而实现了简单、稳定且有效的训练和推理方案。大量实验证明了我们方法的优越性。
英文摘要：
Multimodal-driven talking face generation refers to animating a portrait with the given pose, expression, and gaze transferred from the driving image and video, or estimated from the text and audio. However, existing methods ignore the potential of text modal, and their generators mainly follow the source-oriented feature rearrange paradigm coupled with unstable GAN frameworks. In this work, we first represent the emotion in the text prompt, which could inherit rich semantics from the CLIP, allowing flexible and generalized emotion control. We further reorganize these tasks as the target-oriented texture transfer and adopt the Diffusion Models. More specifically, given a textured face as the source and the rendered face projected from the desired 3DMM coefficients as the target, our proposed Texture-Geometry-aware Diffusion Model decomposes the complex transfer problem into multi-conditional denoising process, where a Texture Attention-based module accurately models the correspondences between appearance and geometry cues contained in source and target conditions, and incorporate extra implicit information for high-fidelity talking face generation. Additionally, TGDM can be gracefully tailored for face swapping. We derive a novel paradigm free of unstable seesaw-style optimization, resulting in simple, stable, and effective training and inference schemes. Extensive experiments demonstrate the superiority of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
