NeRF-AD：面向说话人脸合成的带基于注意力解耦的神经辐射场
NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis
AI总结：
针对现有基于NeRF的说话人脸合成方法唇形不准确的问题，提出带注意力解耦的NeRF-AD方法，通过AU信息解耦人脸并监督音频与音频人脸的融合，在图像质量和唇形同步上优于现有最优方法。
摘要：
音频驱动的说话人脸合成是当前多维信号处理与多媒体领域的研究热点之一。为提升生成人脸的真实感与3D效果，神经辐射场（Neural Radiance Field, NeRF）近期被引入该研究领域。然而，现有大多数基于NeRF的方法要么给NeRF施加了复杂的学习任务，同时缺乏有监督的多模态特征融合方法，要么无法将音频精准映射到与语音运动相关的人脸区域，最终导致现有方法生成的唇形不准确。本文将部分NeRF学习任务前置，提出了一种基于带注意力解耦的NeRF的说话人脸合成方法（NeRF-AD）。具体而言，该方法引入基于注意力的解耦模块，利用与语音相关的面部动作单元（AU）信息，将人脸解耦为音频人脸（Audio-face）和身份人脸（Identity-face）。为精准调控音频对说话人脸的影响，我们仅将音频人脸与音频特征进行融合，同时还利用AU信息对这两种模态的融合过程进行监督。大量定性与定量实验表明，我们的NeRF-AD在生成真实说话人脸视频方面（包括图像质量与唇形同步性）优于现有最优方法。视频结果可参见https://xiaoxingliu02.github.io/NeRF-AD。
英文摘要：
Talking face synthesis driven by audio is one of the current research hotspots in the fields of multidimensional signal processing and multimedia. Neural Radiance Field (NeRF) has recently been brought to this research field in order to enhance the realism and 3D effect of the generated faces. However, most existing NeRF-based methods either burden NeRF with complex learning tasks while lacking methods for supervised multimodal feature fusion, or cannot precisely map audio to the facial region related to speech movements. These reasons ultimately result in existing methods generating inaccurate lip shapes. This paper moves a portion of NeRF learning tasks ahead and proposes a talking face synthesis method via NeRF with attention-based disentanglement (NeRF-AD). In particular, an Attention-based Disentanglement module is introduced to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. To precisely regulate how audio affects the talking face, we only fuse the Audio-face with audio feature. In addition, AU information is also utilized to supervise the fusion of these two modalities. Extensive qualitative and quantitative experiments demonstrate that our NeRF-AD outperforms state-of-the-art methods in generating realistic talking face videos, including image quality and lip synchronization. To view video results, please refer to https://xiaoxingliu02.github.io/NeRF-AD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
