arXiv 2601.148752026-01-21cs.CVcs.AI

GAT-NeRF：基于几何感知的Transformer增强神经辐射场用于高保真的4D面部虚拟人物

GAT-NeRF: Geometry-Aware-Transformer Enhanced Neural Radiance Fields for High-Fidelity 4D Facial Avatars

Department of Control Science and Engineering, University of Shanghai for Science and Technology（控制科学与工程系，上海科学技术大学）；Business School, University of Shanghai for Science and Technology（商学院，上海科学技术大学）；School of Psychology and Neuroscience, University of Glasgow（心理学与神经科学学院，格拉斯哥大学）

Zhe Chang, Haodong Jin, Ying Sun, Yan Song, Hui Yu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

GAT-NeRF通过整合Transformer机制与几何感知模块，提升从单目视频重建高保真4D动态面部虚拟人物的能力。

摘要：

从单目视频重建高保真的4D动态面部虚拟人物是关键但具有挑战性的任务，受日益增长的沉浸式虚拟人类应用需求驱动。尽管神经辐射场（NeRF）已推动场景表示的进步，但其捕捉高频面部细节（如动态皱纹和来自信息受限单目流的细微纹理）的能力需要显著增强。为解决这一挑战，我们提出了一种新的混合神经辐射场框架，称为几何感知Transformer增强NeRF（GAT-NeRF），用于高保真和可控的4D面部虚拟人物重建，该框架将Transformer机制整合到NeRF流程中。GAT-NeRF协同结合了坐标对齐的多层感知机（MLP）和轻量级Transformer模块，称为几何感知Transformer（GAT），因其处理包含显式几何先验的多模态输入。GAT模块通过融合多模态输入特征（包括3D空间坐标、3D形变模型（3DMM）表达参数和可学习的潜在代码）来有效学习和增强与精细几何相关的特征表示。Transformer的有效特征学习能力被用来显著增强复杂局部面部模式（如动态皱纹和痤疮疤痕）的建模。全面的实验明确证明了GAT-NeRF在视觉保真度和高频细节恢复方面的最先进性能，为多媒体应用创建逼真动态数字人类开辟了新途径。

英文摘要：

High-fidelity 4D dynamic facial avatar reconstruction from monocular video is a critical yet challenging task, driven by increasing demands for immersive virtual human applications. While Neural Radiance Fields (NeRF) have advanced scene representation, their capacity to capture high-frequency facial details, such as dynamic wrinkles and subtle textures from information-constrained monocular streams, requires significant enhancement. To tackle this challenge, we propose a novel hybrid neural radiance field framework, called Geometry-Aware-Transformer Enhanced NeRF (GAT-NeRF) for high-fidelity and controllable 4D facial avatar reconstruction, which integrates the Transformer mechanism into the NeRF pipeline. GAT-NeRF synergistically combines a coordinate-aligned Multilayer Perceptron (MLP) with a lightweight Transformer module, termed as Geometry-Aware-Transformer (GAT) due to its processing of multi-modal inputs containing explicit geometric priors. The GAT module is enabled by fusing multi-modal input features, including 3D spatial coordinates, 3D Morphable Model (3DMM) expression parameters, and learnable latent codes to effectively learn and enhance feature representations pertinent to fine-grained geometry. The Transformer's effective feature learning capabilities are leveraged to significantly augment the modeling of complex local facial patterns like dynamic wrinkles and acne scars. Comprehensive experiments unequivocally demonstrate GAT-NeRF's state-of-the-art performance in visual fidelity and high-frequency detail recovery, forging new pathways for creating realistic dynamic digital humans for multimedia applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。