基于NeRF和网格光栅化混合方案的3D人脸风格迁移
3D Face Style Transfer with a Hybrid Solution of NeRF and Mesh Rasterization
AI总结：
本文提出一种结合NeRF和网格光栅化的三阶段混合框架，通过提取网格优化与条件颜色网络，解决了3D人脸风格迁移中的多视角一致性与收敛性问题，实现了高质量且风格可控的3D风格迁移。
摘要：
近年来，人脸风格迁移已被广泛研究。现有的大多数方法在2D图像域中工作，当应用于同一个人脸的不同视角时，存在3D不一致性问题。在本文中，我们解决了3D人脸风格迁移问题，旨在生成具有多视角一致性的3D人脸风格化新视角。我们提出使用神经辐射场（NeRF）来表示3D人脸，并将其与2D风格迁移相结合以对人脸进行风格化。我们发现，直接在2D风格迁移生成的风格化图像上训练NeRF会引入3D不一致性问题并导致模糊。另一方面，将NeRF与2D风格迁移目标联合训练，由于风格图像和内容图像之间存在身份和头部姿态差异，表现出较差的收敛性。此外，由于需要对完整图像进行体渲染以应用风格迁移损失函数，这在训练时间和内存方面也带来了挑战。因此，我们提出了一个NeRF和网格光栅化的混合框架，结合了NeRF高保真几何重建和网格快速渲染的优势。我们的框架包含三个阶段：1. 在输入人脸图像上训练NeRF模型以学习3D几何；2. 从训练好的NeRF模型中提取网格，并通过可微光栅化结合风格迁移目标对其进行优化；3. 在NeRF中训练一个以风格嵌入为条件的新颜色网络，以实现对3D人脸的任意风格迁移。实验结果表明，我们的方法生成了高质量且具有出色3D一致性的人脸风格迁移结果，同时实现了灵活的风格控制。
英文摘要：
Style transfer for human face has been widely researched in recent years. Majority of the existing approaches work in 2D image domain and have 3D inconsistency issue when applied on different viewpoints of the same face. In this paper, we tackle the problem of 3D face style transfer which aims at generating stylized novel views of a 3D human face with multi-view consistency. We propose to use a neural radiance field (NeRF) to represent 3D human face and combine it with 2D style transfer to stylize the 3D face. We find that directly training a NeRF on stylized images from 2D style transfer brings in 3D inconsistency issue and causes blurriness. On the other hand, training a NeRF jointly with 2D style transfer objectives shows poor convergence due to the identity and head pose gap between style image and content image. It also poses challenge in training time and memory due to the need of volume rendering for full image to apply style transfer loss functions. We therefore propose a hybrid framework of NeRF and mesh rasterization to combine the benefits of high fidelity geometry reconstruction of NeRF and fast rendering speed of mesh. Our framework consists of three stages: 1. Training a NeRF model on input face images to learn the 3D geometry; 2. Extracting a mesh from the trained NeRF model and optimizing it with style transfer objectives via differentiable rasterization; 3. Training a new color network in NeRF conditioned on a style embedding to enable arbitrary style transfer to the 3D face. Experiment results show that our approach generates high quality face style transfer with great 3D consistency, while also enabling a flexible style control.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
