AE-NeRF: 面向少样本说话人脸合成的音频增强神经辐射场
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
AI总结：
针对少样本说话人脸合成中现有NeRF方法的先验缺失与区域音频关联度忽略问题，提出带音频感知聚合和双NeRF对齐生成的AE-NeRF，在保真度、唇音同步和泛化性上优于SOTA。
摘要：
音频驱动的说话人脸合成是一个具有广阔应用前景的研究方向，可应用于数字人、电影制作和虚拟现实等领域。近年来，基于神经辐射场（NeRF）的方法相较于过往研究在生成质量和逼真度上展现出了优势。然而，在少样本说话人脸生成这一实际场景中（即仅能获取某一身份数秒的说话视频），现有方法存在两点局限：1）它们要么没有可作为面部先验以实现快速收敛的基础模型，要么在构建先验时忽略了音频的重要性；2）大多数方法忽视了不同面部区域与音频的关联程度差异，例如嘴部与音频相关，而耳部则与音频无关。本文提出音频增强神经辐射场（Audio Enhanced Neural Radiance Field，AE-NeRF）以解决上述问题，该方法能够利用少样本数据集生成新说话者的逼真人像。具体而言，我们在参考方案的特征融合阶段引入了音频感知聚合模块，其权重由参考图像与目标图像之间的音频相似度决定。随后，我们提出了音频对齐人脸生成策略，采用双NeRF框架分别对音频相关区域和音频无关区域进行建模。大量实验表明，即使在训练集有限或训练迭代次数较少的情况下，AE-NeRF在图像保真度、音频-唇部同步性以及泛化能力方面均超越了现有最优方法。
英文摘要：
Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality. Recent NeRF-based approaches have shown superiority in quality and fidelity compared to previous studies. However, when it comes to few-shot talking head generation, a practical scenario where only few seconds of talking video is available for one identity, two limitations emerge: 1) they either have no base model, which serves as a facial prior for fast convergence, or ignore the importance of audio when building the prior; 2) most of them overlook the degree of correlation between different face regions and audio, e.g., mouth is audio related, while ear is audio independent. In this paper, we present Audio Enhanced Neural Radiance Field (AE-NeRF) to tackle the above issues, which can generate realistic portraits of a new speaker with fewshot dataset. Specifically, we introduce an Audio Aware Aggregation module into the feature fusion stage of the reference scheme, where the weight is determined by the similarity of audio between reference and target image. Then, an Audio-Aligned Face Generation strategy is proposed to model the audio related and audio independent regions respectively, with a dual-NeRF framework. Extensive experiments have shown AE-NeRF surpasses the state-of-the-art on image fidelity, audio-lip synchronization, and generalization ability, even in limited training set or training iterations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
