面向音频驱动面部重演的参数化隐式人脸表征
Parametric Implicit Face Representation for Audio-Driven Facial Reenactment
AI总结：
针对音频驱动面部重演中显式与隐式表征的权衡问题，提出参数化隐式人脸表征及可控高质量重演框架，通过多项技术优化提升生成效果与泛化性，实验验证其在逼真度和身份风格保真度上优于现有方法。
摘要：
音频驱动面部重演是一项关键技术，在电影制作、虚拟化身和视频会议领域有广泛应用。现有研究要么采用显式中间人脸表征（如2D人脸关键点或3D人脸模型），要么采用隐式表征（如神经辐射场Neural Radiance Fields），因此存在可解释性与表达能力之间的权衡，进而导致结果的可控性与质量难以兼顾。在本研究中，我们通过提出的新型参数化隐式人脸表征打破了这类权衡，并构建了一个兼具可控性、能够生成高质量说话人头的音频驱动面部重演框架。具体而言，我们的参数化隐式表征利用3D人脸模型的可解释参数对隐式表征进行参数化，从而兼具显式方法与隐式方法的优势。此外，我们提出了多项新技术来优化框架的三个组成部分：其一，将上下文信息融入音频到表情参数的编码过程；其二，采用条件图像合成实现隐式表征的参数化，并通过创新的三平面结构实现该模块以提升学习效率；其三，将面部重演建模为条件图像修复问题，并提出一种新型数据增强技术以提升模型的泛化能力。大量实验表明，与现有方法相比，我们的方法能生成更逼真的结果，且对说话者的身份和说话风格具有更高的保真度。
英文摘要：
Audio-driven facial reenactment is a crucial technique that has a range of applications in film-making, virtual avatars and video conferences. Existing works either employ explicit intermediate face representations (e.g., 2D facial landmarks or 3D face models) or implicit ones (e.g., Neural Radiance Fields), thus suffering from the trade-offs between interpretability and expressive power, hence between controllability and quality of the results. In this work, we break these trade-offs with our novel parametric implicit face representation and propose a novel audio-driven facial reenactment framework that is both controllable and can generate high-quality talking heads. Specifically, our parametric implicit representation parameterizes the implicit representation with interpretable parameters of 3D face models, thereby taking the best of both explicit and implicit methods. In addition, we propose several new techniques to improve the three components of our framework, including i) incorporating contextual information into the audio-to-expression parameters encoding; ii) using conditional image synthesis to parameterize the implicit representation and implementing it with an innovative tri-plane structure for efficient learning; iii) formulating facial reenactment as a conditional image inpainting problem and proposing a novel data augmentation technique to improve model generalizability. Extensive experiments demonstrate that our method can generate more realistic results than previous methods with greater fidelity to the identities and talking styles of speakers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
