FD2Talk：面向通用化说话人头生成的人脸解耦扩散模型
FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model
AI总结：
针对现有说话人头生成方法质量不足、扩散模型应用存在局限的问题，提出FD2Talk人脸解耦扩散模型，通过两阶段解耦运动与外观，提升了生成质量与结果的准确性、多样性。
摘要：
说话人头生成是一项重要的研究课题，目前仍面临诸多挑战。以往的研究通常采用生成对抗网络或回归模型，这类方法受困于生成质量不佳以及平均人脸形状问题。尽管扩散模型展现出了出色的生成能力，但它们在说话人头生成领域的探索仍不尽人意。究其原因，要么是仅利用扩散模型获取中间表示，再借助另一个预训练渲染器完成生成，要么是忽略了复杂人脸细节（如表情、头部姿态和外观纹理）的特征解耦。为此，我们提出了一种用于说话人头生成的人脸解耦扩散模型，名为FD2Talk，该模型充分发挥扩散模型的优势，通过多阶段对复杂人脸细节进行解耦。具体而言，我们将人脸细节拆分为运动与外观两部分。在第一阶段，我们设计了Diffusion Transformer（扩散Transformer），从原始音频中准确预测运动系数。这些运动与外观高度解耦，相较于高维RGB图像，更易于网络学习。随后在第二阶段，我们对参考图像进行编码，以提取外观纹理信息。预测得到的人脸与头部运动，以及编码后的外观，将作为Diffusion UNet（扩散UNet）的条件，引导视频帧的生成。得益于人脸细节解耦以及对扩散模型的充分利用，大量实验证实，与现有的最优方法相比，我们的方法在提升图像质量方面表现更优，且能生成更准确、更多样化的结果。
英文摘要：
Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they either solely use the diffusion model to obtain an intermediate representation and then employ another pre-trained renderer, or they overlook the feature decoupling of complex facial details, such as expressions, head poses and appearance textures. Therefore, we propose a Facial Decoupled Diffusion model for Talking head generation called FD2Talk, which fully leverages the advantages of diffusion models and decouples the complex facial details through multi-stages. Specifically, we separate facial details into motion and appearance. In the initial phase, we design the Diffusion Transformer to accurately predict motion coefficients from raw audio. These motions are highly decoupled from appearance, making them easier for the network to learn compared to high-dimensional RGB images. Subsequently, in the second phase, we encode the reference image to capture appearance textures. The predicted facial and head motions and encoded appearance then serve as the conditions for the Diffusion UNet, guiding the frame generation. Benefiting from decoupling facial details and fully leveraging diffusion models, extensive experiments substantiate that our approach excels in enhancing image quality and generating more accurate and diverse results compared to previous state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
