NeRF-3DTalker：用于说话人头部合成的、具备3D先验辅助音频解耦的神经辐射场
NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis
AI总结：
本文提出NeRF-3DTalker，利用3D先验、音频解耦模块和局部-全局标准化空间，实现自由视角、逼真且唇形同步的3D说话人头部合成。
摘要：
说话人头部合成旨在利用音频合成唇形同步的说话人头部视频。近年来，NeRF增强所合成说话人头部真实感和纹理细节的能力引起了研究人员的关注。然而，当前大多数基于音频的NeRF方法只关注正面人脸的渲染。这些方法无法在新视角下生成清晰的说话人头部。当前3D说话人头部合成中另一个普遍挑战是难以对齐声学空间与视觉空间，这往往导致生成的说话人头部唇形同步效果欠佳。为解决这些问题，我们提出了用于说话人头部合成的、具备3D先验辅助音频解耦的神经辐射场（NeRF-3DTalker）。具体而言，所提出的方法利用3D先验信息合成具有自由视角的清晰说话人头部。此外，我们提出了一个3D先验辅助音频解耦模块，该模块旨在将音频解耦为两个不同类别：与3D赋予的语音运动相关的特征，以及与说话风格相关的特征。此外，为了将远离说话人运动空间的生成帧重新定位到真实空间中，我们设计了一个局部-全局标准化空间。该方法从全局和局部语义两个角度对生成帧中的不规则位置进行归一化。通过全面的定性和定量实验，结果表明我们的NeRF-3DTalker在合成逼真的说话人头部视频方面优于最先进方法，表现出更优的图像质量和唇形同步效果。项目页面：https://nerf-3dtalker.github.io/NeRF-3Dtalker。
英文摘要：
Talking head synthesis is to synthesize a lip-synchronized talking head video using audio. Recently, the capability of NeRF to enhance the realism and texture details of synthesized talking heads has attracted the attention of researchers. However, most current NeRF methods based on audio are exclusively concerned with the rendering of frontal faces. These methods are unable to generate clear talking heads in novel views. Another prevalent challenge in current 3D talking head synthesis is the difficulty in aligning acoustic and visual spaces, which often results in suboptimal lip-syncing of the generated talking heads. To address these issues, we propose Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesis (NeRF-3DTalker). Specifically, the proposed method employs 3D prior information to synthesize clear talking heads with free views. Additionally, we propose a 3D Prior Aided Audio Disentanglement module, which is designed to disentangle the audio into two distinct categories: features related to 3D awarded speech movements and features related to speaking style. Moreover, to reposition the generated frames that are distant from the speaker's motion space in the real space, we have devised a local-global Standardized Space. This method normalizes the irregular positions in the generated frames from both global and local semantic perspectives. Through comprehensive qualitative and quantitative experiments, it has been demonstrated that our NeRF-3DTalker outperforms state-of-the-art in synthesizing realistic talking head videos, exhibiting superior image quality and lip synchronization. Project page: https://nerf-3dtalker.github.io/NeRF-3Dtalker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
