DiffPortrait3D：面向零样本人像视图合成的可控扩散模型
DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis
AI总结：
本研究提出DiffPortrait3D条件扩散模型，利用2D扩散先验结合解耦控制、跨视角注意力及3D感知噪声生成，实现单张人像零样本3D一致新视图合成，在多基准上达SOTA效果。
摘要：
我们提出了DiffPortrait3D，这是一种条件扩散模型，能够从少至一张真实场景下的人像合成具有3D一致性的逼真新视图。具体而言，给定单张RGB输入，我们的目标是合成从新相机视角渲染的合理且一致的面部细节，同时保留身份和面部表情。无需耗时的优化和微调，我们的零样本方法能很好地泛化到任意人脸人像，适配未标定相机视角、极端面部表情以及多样的艺术表现形式。其核心在于，我们利用在大规模图像数据集上预训练的2D扩散模型的生成先验作为渲染骨干，同时通过外观与相机姿态的解耦注意力控制来引导去噪过程。为此，我们首先将参考图像的外观上下文注入到冻结UNet的自注意力层中。随后，通过一个新颖的条件控制模块操控渲染视角，该模块通过观察同一视角下交叉主体的条件图像来解析相机姿态。此外，我们插入了一个可训练的跨视角注意力模块以增强视图一致性，并在推理过程中通过一种新颖的3D感知噪声生成过程进一步强化这种一致性。我们在具有挑战性的真实场景和多视图基准上，从定性和定量两方面展示了最先进的结果。
英文摘要：
We present DiffPortrait3D, a conditional diffusion model that is capable of synthesizing 3D-consistent photo-realistic novel views from as few as a single in-the-wild portrait. Specifically, given a single RGB input, we aim to synthesize plausible but consistent facial details rendered from novel camera views with retained both identity and facial expression. In lieu of time-consuming optimization and fine-tuning, our zero-shot method generalizes well to arbitrary face portraits with unposed camera views, extreme facial expressions, and diverse artistic depictions. At its core, we leverage the generative prior of 2D diffusion models pre-trained on large-scale image datasets as our rendering backbone, while the denoising is guided with disentangled attentive control of appearance and camera pose. To achieve this, we first inject the appearance context from the reference image into the self-attention layers of the frozen UNets. The rendering view is then manipulated with a novel conditional control module that interprets the camera pose by watching a condition image of a crossed subject from the same view. Furthermore, we insert a trainable cross-view attention module to enhance view consistency, which is further strengthened with a novel 3D-aware noise generation process during inference. We demonstrate state-of-the-art results both qualitatively and quantitatively on our challenging in-the-wild and multi-view benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
