NeRFFaceSpeech：通过生成先验实现单次音频驱动的3D说话头部合成
NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Prior
AI总结：
本文提出NeRFFaceSpeech，结合生成先验与NeRF，利用参数化面部模型和LipaintNet实现单图音频驱动的高质量3D一致说话头部合成，并首次量化姿态鲁棒性。
摘要：
音频驱动的说话头部生成正从2D内容向3D内容发展。值得注意的是，神经辐射场（Neural Radiance Field，NeRF）作为一种合成高质量3D说话头部输出的手段正受到关注。遗憾的是，这种基于NeRF的方法通常需要为每个身份准备大量配对的音视频数据，从而限制了该方法的可扩展性。尽管已有尝试利用单张图像生成音频驱动的3D说话头部动画，但由于图像中被遮挡区域的信息不足，结果往往不能令人满意。本文主要关注在单次、音频驱动领域中被忽视的3D一致性问题，该领域中的面部动画主要在正面视角下合成。我们提出了一种名为NeRFFaceSpeech的新方法，能够生成高质量的具备3D感知能力的说话头部。通过结合生成模型的先验知识与NeRF，我们的方法能够构建与单张图像相对应的3D一致面部特征空间。我们的空间同步方法采用参数化面部模型中与音频相关的顶点动态，通过光线变形将静态图像特征转换为动态视觉内容，从而确保逼真的3D面部运动。此外，我们提出了LipaintNet，它可以补充口腔内部区域缺失的信息，而这些信息无法从给定的单张图像中获得。该网络以自监督方式训练，利用生成能力而无需额外数据。综合实验表明，与以往方法相比，我们的方法在从单张图像生成音频驱动说话头部方面具有优越性，并增强了3D一致性。此外，我们首次提出了一种定量衡量模型对姿态变化鲁棒性的方法，而此前这只能通过定性方式完成。
英文摘要：
Audio-driven talking head generation is advancing from 2D to 3D content. Notably, Neural Radiance Field (NeRF) is in the spotlight as a means to synthesize high-quality 3D talking head outputs. Unfortunately, this NeRF-based approach typically requires a large number of paired audio-visual data for each identity, thereby limiting the scalability of the method. Although there have been attempts to generate audio-driven 3D talking head animations with a single image, the results are often unsatisfactory due to insufficient information on obscured regions in the image. In this paper, we mainly focus on addressing the overlooked aspect of 3D consistency in the one-shot, audio-driven domain, where facial animations are synthesized primarily in front-facing perspectives. We propose a novel method, NeRFFaceSpeech, which enables to produce high-quality 3D-aware talking head. Using prior knowledge of generative models combined with NeRF, our method can craft a 3D-consistent facial feature space corresponding to a single image. Our spatial synchronization method employs audio-correlated vertex dynamics of a parametric face model to transform static image features into dynamic visuals through ray deformation, ensuring realistic 3D facial motion. Moreover, we introduce LipaintNet that can replenish the lacking information in the inner-mouth area, which can not be obtained from a given single image. The network is trained in a self-supervised manner by utilizing the generative capabilities without additional data. The comprehensive experiments demonstrate the superiority of our method in generating audio-driven talking heads from a single image with enhanced 3D consistency compared to previous approaches. In addition, we introduce a quantitative way of measuring the robustness of a model against pose changes for the first time, which has been possible only qualitatively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
