用于高保真说话人像合成的高效区域感知神经辐射场
Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis
AI总结：
本文提出ER-NeRF架构，通过三平面哈希表示、区域注意力模块和自适应姿态编码，实现高效的高保真说话人像合成，兼顾收敛速度、渲染效率和生成质量。
摘要：
本文提出ER-NeRF，这是一种基于条件神经辐射场（Neural Radiance Fields, NeRF）的新型架构，用于说话人像合成，可同时实现快速收敛、实时渲染，且模型体积小、性能达到当前最优水平。我们的核心思路是明确利用空间区域贡献的不均衡性来指导说话人像建模。具体而言，为提升动态头部重建的精度，本文引入了一种紧凑且表达能力强的基于NeRF的三平面哈希表示（Tri-Plane Hash Representation），通过三个平面哈希编码器剪枝空的空间区域。针对语音音频，我们提出区域注意力模块（Region Attention Module），通过注意力机制生成区域感知的条件特征。现有方法多采用基于MLP的编码器隐式学习跨模态关系，与之不同，该注意力机制在音频特征与空间区域之间建立显式关联，以捕捉局部运动先验。此外，本文还引入了直接且快速的自适应姿态编码（Adaptive Pose Encoding），通过将头部姿态的复杂变换映射到空间坐标，优化头躯干分离问题。大量实验表明，与现有方法相比，我们的方法能渲染出质量更高、音唇同步的高保真说话人像视频，细节逼真且效率更高。
英文摘要：
This paper presents ER-NeRF, a novel conditional Neural Radiance Fields (NeRF) based architecture for talking portrait synthesis that can concurrently achieve fast convergence, real-time rendering, and state-of-the-art performance with small model size. Our idea is to explicitly exploit the unequal contribution of spatial regions to guide talking portrait modeling. Specifically, to improve the accuracy of dynamic head reconstruction, a compact and expressive NeRF-based Tri-Plane Hash Representation is introduced by pruning empty spatial regions with three planar hash encoders. For speech audio, we propose a Region Attention Module to generate region-aware condition feature via an attention mechanism. Different from existing methods that utilize an MLP-based encoder to learn the cross-modal relation implicitly, the attention mechanism builds an explicit connection between audio features and spatial regions to capture the priors of local motions. Moreover, a direct and fast Adaptive Pose Encoding is introduced to optimize the head-torso separation problem by mapping the complex transformation of the head pose into spatial coordinates. Extensive experiments demonstrate that our method renders better high-fidelity and audio-lips synchronized talking portrait videos, with realistic details and high efficiency compared to previous methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
