GeneFace++：通用且稳定的实时音频驱动3D数字人说话面部生成
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
AI总结：
针对基于NeRF的音频驱动3D说话面部生成在唇同步、视频质量、效率上的挑战，提出GeneFace++，通过音高辅助与时间损失、landmark局部线性嵌入、高效NeRF渲染器，实现稳定实时的通用音唇同步生成，效果优于现有最优基线。
摘要：
利用任意语音音频生成说话人物肖像，是数字人与元宇宙领域的关键问题。现代说话面部生成方法需要实现通用的音唇同步、优良的视频质量以及高系统效率。近年来，神经辐射场（Neural Radiance Field，NeRF）已成为该领域流行的渲染技术，因为它仅需数分钟的训练视频，就能生成高保真、3D一致的说话面部。然而，基于NeRF的方法仍存在若干挑战：1）在唇形同步方面，难以生成具有高时间一致性和音唇准确率的长面部运动序列；2）在视频质量方面，由于训练渲染器的数据有限，该类方法易受域外输入条件影响，偶尔会产生较差的渲染结果；3）在系统效率方面，原始NeRF缓慢的训练和推理速度严重阻碍了其在实际应用中的使用。本文提出GeneFace++以应对这些挑战：1）将音高轮廓作为辅助特征，并在面部运动预测过程中引入时间损失；2）提出一种landmark局部线性嵌入方法，对预测运动序列中的异常值进行规整，以避免鲁棒性问题；3）设计一种计算高效的基于NeRF的运动到视频渲染器，以实现快速训练和实时推理。通过这些设置，GeneFace++成为首个实现稳定实时说话面部生成、且具备通用音唇同步能力的基于NeRF的方法。大量实验表明，我们的方法在主观和客观评估上均优于现有最优基线。视频样本可访问https://genefaceplusplus.github.io 查看。
英文摘要：
Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. However, there still exist several challenges for NeRF-based methods: 1) as for the lip synchronization, it is hard to generate a long facial motion sequence of high temporal consistency and audio-lip accuracy; 2) as for the video quality, due to the limited data used to train the renderer, it is vulnerable to out-of-domain input condition and produce bad rendering results occasionally; 3) as for the system efficiency, the slow training and inference speed of the vanilla NeRF severely obstruct its usage in real-world applications. In this paper, we propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process; 2) proposing a landmark locally linear embedding method to regulate the outliers in the predicted motion sequence to avoid robustness issues; 3) designing a computationally efficient NeRF-based motion-to-video renderer to achieves fast training and real-time inference. With these settings, GeneFace++ becomes the first NeRF-based method that achieves stable and real-time talking face generation with generalized audio-lip synchronization. Extensive experiments show that our method outperforms state-of-the-art baselines in terms of subjective and objective evaluation. Video samples are available at https://genefaceplusplus.github.io .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
