Cafca：从少量随意拍摄中高质量合成表情丰富人脸的新视角
Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures
AI总结：
Cafca利用仅在合成数据上训练的条件神经辐射场先验，结合三维可变形人脸模型，仅需三张真实图像微调即可实现高保真表情人脸建模与新视角合成，超越现有稀疏输入方法。
摘要：
体积建模和神经辐射场表示已经彻底改变了三维人脸捕捉和照片级真实感新视角合成。然而，这些方法通常需要数百张多视角输入图像，因此不适用于输入不足几张的情况。我们提出了一种新颖的人脸体积先验，仅需在自然环境中拍摄的三个输入视角，即可实现高保真度的表情丰富人脸建模。我们的关键见解是，仅在合成数据上训练的隐式先验可以泛化到极具挑战性的真实世界身份和表情，并渲染出具有皱纹和睫毛等精细个性化细节的新视角。我们利用三维可变形人脸模型合成大规模训练集，为每个身份渲染不同表情、头发、服装和其他资产。然后，我们在这个合成数据集上训练条件神经辐射场先验，并在推理时在单个对象的极稀疏真实图像集上微调模型。平均而言，微调仅需三个输入即可跨越合成到真实的域差距。由此产生的个性化三维模型重建出强烈的个性化面部表情，并在稀疏输入下高质量人脸新视角合成的感知质量和光度质量方面优于现有最先进方法。
英文摘要：
Volumetric modeling and neural radiance field representations have revolutionized 3D face capture and photorealistic novel view synthesis. However, these methods often require hundreds of multi-view input images and are thus inapplicable to cases with less than a handful of inputs. We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions, hair, clothing, and other assets. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject. On average, the fine-tuning requires only three inputs to cross the synthetic-to-real domain gap. The resulting personalized 3D model reconstructs strong idiosyncratic facial expressions and outperforms the state-of-the-art in high-quality novel view synthesis of faces from sparse inputs in terms of perceptual and photo-metric quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
