基于可变形神经辐射场的一次性高保真说话头合成
One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field
AI总结：
本文提出HiDe-NeRF，基于可变形神经辐射场将三维动态人脸解耦为规范外观场与隐式变形场，并通过多尺度体积特征和姿态-表情解耦模块实现一次性高保真、自由视角的说话头合成。
摘要：
说话头生成旨在生成既保持源图像身份信息又模仿驱动图像动作的人脸。大多数开创性方法主要依赖二维表示，因此在遇到大幅度头部旋转时不可避免地会出现人脸失真。近期工作转而采用显式三维结构表示或隐式神经渲染，以改善大姿态变化下的性能。然而，身份和表情的保真度仍不理想，尤其是在新视角合成方面。本文提出HiDe-NeRF，实现高保真、自由视角的说话头合成。借鉴最近提出的可变形神经辐射场，HiDe-NeRF将三维动态场景表示为规范外观场和隐式变形场，其中前者包含规范源人脸，后者对驱动姿态和表情进行建模。具体而言，我们从两个方面提升保真度：(i) 为增强身份表现力，设计了一个通用外观模块，利用多尺度体积特征来保持人脸形状和细节；(ii) 为提高表情精确性，提出一个轻量级变形模块，显式解耦姿态和表情，以实现精确的表情建模。大量实验表明，所提方法能够生成优于以往工作的结果。项目主页：https://www.waytron.net/hidenerf/
英文摘要：
Talking head generation aims to generate faces that maintain the identity information of the source image and imitate the motion of the driving image. Most pioneering methods rely primarily on 2D representations and thus will inevitably suffer from face distortion when large head rotations are encountered. Recent works instead employ explicit 3D structural representations or implicit neural rendering to improve performance under large pose changes. Nevertheless, the fidelity of identity and expression is not so desirable, especially for novel-view synthesis. In this paper, we propose HiDe-NeRF, which achieves high-fidelity and free-view talking-head synthesis. Drawing on the recently proposed Deformable Neural Radiance Fields, HiDe-NeRF represents the 3D dynamic scene into a canonical appearance field and an implicit deformation field, where the former comprises the canonical source face and the latter models the driving pose and expression. In particular, we improve fidelity from two aspects: (i) to enhance identity expressiveness, we design a generalized appearance module that leverages multi-scale volume features to preserve face shape and details; (ii) to improve expression preciseness, we propose a lightweight deformation module that explicitly decouples the pose and expression to enable precise expression modeling. Extensive experiments demonstrate that our proposed approach can generate better results than previous works. Project page: https://www.waytron.net/hidenerf/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
