Talk3D：基于个性化3D生成先验的高保真说话人像合成
Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D Generative Prior
AI总结：
本文提出Talk3D框架，通过预训练3D感知生成先验和音频引导注意力U-Net，在NeRF空间中重建高保真面部几何，并利用音频无关条件令牌解耦无关变化，在极端姿态下仍优于现有方法。
摘要：
近年来，音频驱动的说话头像合成方法通常在单目说话人像视频上优化神经辐射场（NeRF），利用其渲染高保真且3D一致的新视角帧的能力。然而，由于输入的单目视频缺乏全面的3D信息，这些方法往往难以重建完整的面部几何结构。本文提出了一种新颖的音频驱动说话头像合成框架Talk3D，通过有效采用预训练的3D感知生成先验，能够忠实地重建合理的面部几何结构。在给定个性化3D生成模型的基础上，我们提出了一种新颖的音频引导注意力U-Net架构，用于在音频驱动下预测NeRF空间中的动态面部变化。此外，我们的模型还通过音频无关的条件令牌进行调制，从而有效解耦与音频特征无关的变化。与现有方法相比，我们的方法即使在极端头部姿态下也能生成逼真的面部几何结构。我们还进行了大量实验，表明我们的方法在定量和定性评估方面均超越了最先进的基准。
英文摘要：
Recent methods for audio-driven talking head synthesis often optimize neural radiance fields (NeRF) on a monocular talking portrait video, leveraging its capability to render high-fidelity and 3D-consistent novel-view frames. However, they often struggle to reconstruct complete face geometry due to the absence of comprehensive 3D information in the input monocular videos. In this paper, we introduce a novel audio-driven talking head synthesis framework, called Talk3D, that can faithfully reconstruct its plausible facial geometries by effectively adopting the pre-trained 3D-aware generative prior. Given the personalized 3D generative model, we present a novel audio-guided attention U-Net architecture that predicts the dynamic face variations in the NeRF space driven by audio. Furthermore, our model is further modulated by audio-unrelated conditioning tokens which effectively disentangle variations unrelated to audio features. Compared to existing methods, our method excels in generating realistic facial geometries even under extreme head poses. We also conduct extensive experiments showing our approach surpasses state-of-the-art benchmarks in terms of both quantitative and qualitative evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
