单目身份条件化的人脸反射率重建
Monocular Identity-Conditioned Facial Reflectance Reconstruction
AI总结：
本文提出ID2Reflectance框架，通过图像空间反射率先验、多域码本融合和身份条件化交换模块，仅用有限反射率数据即可从单张图像重建高质量人脸反射率，并在自然场景中达到最先进性能。
摘要：
近年来，三维人脸重建方法取得了显著进展，但单目高质量人脸反射率重建仍面临巨大挑战。现有方法依赖大量光场采集数据来学习人脸反射率模型，然而受试者多样性的缺乏给实现良好泛化能力和广泛适用性带来了困难。本文在图像空间而非UV空间中学习反射率先验，并提出了一个名为ID2Reflectance的框架。该框架能够在仅使用有限反射率数据进行训练的情况下，直接估计单张图像的反射率图。我们的核心见解是：反射率数据与RGB人脸共享人脸结构，这使得能够从低成本的RGB数据中获得富有表现力的人脸先验，从而减少对反射率数据的依赖。我们首先学习高质量的人脸反射率先验。具体而言，我们预训练多域人脸特征码本，并设计了一种码本融合方法以对齐反射率域和RGB域。然后，我们提出了一个身份条件化交换模块，将目标图像中的人脸身份注入预训练的自编码器，以修改源反射率图像的身份。最后，我们拼接多视角交换后的反射率图像以获得可渲染资产。大量实验表明，我们的方法展现出优异的泛化能力，并在自然场景人脸反射率重建任务上取得了最先进的结果。我们的项目主页为https://xingyuren.github.io/id2reflectance/。
英文摘要：
Recent 3D face reconstruction methods have made remarkable advancements, yet there remain huge challenges in monocular high-quality facial reflectance reconstruction. Existing methods rely on a large amount of light-stage captured data to learn facial reflectance models. However, the lack of subject diversity poses challenges in achieving good generalization and widespread applicability. In this paper, we learn the reflectance prior in image space rather than UV space and present a framework named ID2Reflectance. Our framework can directly estimate the reflectance maps of a single image while using limited reflectance data for training. Our key insight is that reflectance data shares facial structures with RGB faces, which enables obtaining expressive facial prior from inexpensive RGB data thus reducing the dependency on reflectance data. We first learn a high-quality prior for facial reflectance. Specifically, we pretrain multi-domain facial feature codebooks and design a codebook fusion method to align the reflectance and RGB domains. Then, we propose an identity-conditioned swapping module that injects facial identity from the target image into the pre-trained autoencoder to modify the identity of the source reflectance image. Finally, we stitch multi-view swapped reflectance images to obtain renderable assets. Extensive experiments demonstrate that our method exhibits excellent generalization capability and achieves state-of-the-art facial reflectance reconstruction results for in-the-wild faces. Our project page is https://xingyuren.github.io/id2reflectance/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
