基于带人脸先验的语音条件潜扩散模型的真实感人脸生成方法
Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Prior
AI总结：
针对现有GAN-based语音人脸生成方法稳定性差、真实感不足的问题，提出带人脸先验的SCLDM框架，通过对比预训练对齐跨模态属性、引入统计人脸先残差控制多样性，在两个基准数据集上实现身份保留与真实感的显著提升。
摘要：
语音驱动人脸生成是一个颇具吸引力的研究方向，聚焦于根据说话人的音频语音生成真实的人脸图像。然而，现有采用基于GAN（生成对抗网络）架构的最优方法存在稳定性不足的问题，无法生成真实感强的人脸图像。为填补这一空白，我们提出了一种新颖的语音驱动人脸生成框架，该框架利用名为SCLDM的语音条件潜扩散模型（Speech-Conditioned Latent Diffusion Model）。据我们所知，这是首个将扩散模型的卓越建模能力应用于语音驱动人脸生成的研究。保留语音与人脸之间共享的身份信息对于生成真实的结果至关重要，因此我们对语音编码器和人脸编码器均采用对比预训练策略。该预训练策略能够有效对齐语音的年龄、性别等属性与人脸图像中对应的面部特征。此外，我们解决了扩散模型在合成过程中带来的多样性过度的挑战。为克服这一难题，我们引入残差概念，将统计人脸先验整合到扩散过程中。这一设计有助于消除人脸之间的共享成分，增强语音条件所捕捉的细微差异。大量定量、定性实验以及用户研究表明，与现有最优方法相比，我们的方法能够生成更真实的人脸图像，同时更好地保留说话人的身份信息。值得关注的性能提升体现在，我们的方法在AVSpeech数据集和Voxceleb数据集的所有指标上均取得显著增益，其中两个数据集的余弦距离指标分别提升了32.17和32.72，表现尤为突出。
英文摘要：
Speech-to-face generation is an intriguing area of research that focuses on generating realistic facial images based on a speaker's audio speech. However, state-of-the-art methods employing GAN-based architectures lack stability and cannot generate realistic face images. To fill this gap, we propose a novel speech-to-face generation framework, which leverages a Speech-Conditioned Latent Diffusion Model, called SCLDM. To the best of our knowledge, this is the first work to harness the exceptional modeling capabilities of diffusion models for speech-to-face generation. Preserving the shared identity information between speech and face is crucial in generating realistic results. Therefore, we employ contrastive pre-training for both the speech encoder and the face encoder. This pre-training strategy facilitates effective alignment between the attributes of speech, such as age and gender, and the corresponding facial characteristics in the face images. Furthermore, we tackle the challenge posed by excessive diversity in the synthesis process caused by the diffusion model. To overcome this challenge, we introduce the concept of residuals by integrating a statistical face prior to the diffusion process. This addition helps to eliminate the shared component across the faces and enhances the subtle variations captured by the speech condition. Extensive quantitative, qualitative, and user study experiments demonstrate that our method can produce more realistic face images while preserving the identity of the speaker better than state-of-the-art methods. Highlighting the notable enhancements, our method demonstrates significant gains in all metrics on the AVSpeech dataset and Voxceleb dataset, particularly noteworthy are the improvements of 32.17 and 32.72 on the cosine distance metric for the two datasets, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
