S^3D-NeRF：用于高保真说话人头合成的单样本语音驱动神经辐射场
S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis
AI总结：
针对现有NeRF类说话人头合成方法无法直接用音频驱动的问题，提出S^3D-NeRF方法，通过分层外观编码器、跨模态形变场及唇同步判别器，在视频保真度和音唇同步性上优于现有方法。
摘要：
说话人头合成是一项具备广泛应用的实用技术。当前基于神经辐射场（Neural Radiance Field, NeRF）的方法已展现出在利用视频或从音频回归的信号驱动单样本说话人头方面的优势。然而，这类方法大多无法直接将音频作为驱动信息，难以发挥语音的灵活性与易用性。鉴于将音频信号映射到面部形变并非易事，本文设计了单样本语音驱动神经辐射场（Single-Shot Speech-Driven Neural Radiance Field, S^3D-NeRF）方法，以解决三大难点：为每个身份学习具有代表性的外观特征、利用音频建模不同面部区域的运动、保持唇部区域的时间一致性。为此，我们引入分层面部外观编码器（Hierarchical Facial Appearance Encoder）学习多尺度表征，以捕捉不同说话者的外观；并精心设计跨模态面部形变场（Cross-modal Facial Deformation Field），依据音频信号与不同面部区域的关联生成语音驱动的动画。此外，为提升关键唇部区域的时间一致性，我们引入唇形同步判别器，对音视频不同步的序列施加惩罚。大量实验表明，我们的S^3D-NeRF在视频保真度与音唇同步性上均超越了现有技术。
英文摘要：
Talking head synthesis is a practical technique with wide applications. Current Neural Radiance Field (NeRF) based approaches have shown their superiority on driving one-shot talking heads with videos or signals regressed from audio. However, most of them failed to take the audio as driven information directly, unable to enjoy the flexibility and availability of speech. Since mapping audio signals to face deformation is non-trivial, we design a Single-Shot Speech-Driven Neural Radiance Field (S^3D-NeRF) method in this paper to tackle the following three difficulties: learning a representative appearance feature for each identity, modeling motion of different face regions with audio, and keeping the temporal consistency of the lip area. To this end, we introduce a Hierarchical Facial Appearance Encoder to learn multi-scale representations for catching the appearance of different speakers, and elaborate a Cross-modal Facial Deformation Field to perform speech animation according to the relationship between the audio signal and different face regions. Moreover, to enhance the temporal consistency of the important lip area, we introduce a lip-sync discriminator to penalize the out-of-sync audio-visual sequences. Extensive experiments have shown that our S^3D-NeRF surpasses previous arts on both video fidelity and audio-lip synchronization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
