VOODOO 3D：用于单样本3D头部重演的体积化人像解耦
VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment
AI总结：
针对单样本3D头部重演中的身份泄漏与表情不自然问题，提出基于三平面体积表示和自监督学习的实时解耦框架，实现高保真、任意视角的SOTA重演效果。
摘要：
我们提出了一种具备3D感知能力的单样本头部重演方法，该方法基于一个完全体积化的神经解耦框架，用于分离源外观与驱动者表情。我们的方法可实时运行，并生成高保真且视角一致的输出，适用于基于全息显示器的3D远程会议系统。现有前沿的3D感知重演方法通常使用神经辐射场或3D网格来产生视角一致的外观编码，但与此同时，它们依赖3DMM等线性人脸模型来实现外观与面部表情的解耦。因此，其重演结果往往表现出来自驱动者的身份泄漏，或出现不自然的表情。为解决这些问题，我们提出了一种神经自监督解耦方法，将源图像和驱动视频帧均提升到基于三平面的共享3D体积表示中。随后，该表示可利用从驱动图像中提取的表情三平面进行自由操控，并使用神经辐射场从任意视角渲染。我们通过在大型野外视频数据集上进行自监督学习来实现这种解耦。我们还进一步引入了一种高效的微调方法，利用相同的真实世界数据提升3D提升过程的泛化能力。我们在广泛的数据集上展示了最先进的性能，并在极具挑战性和多样性的对象上展示了高质量的3D感知头部重演，包括源对象和驱动者均具有非正面头部姿态和复杂表情的情形。
英文摘要：
We present a 3D-aware one-shot head reenactment method based on a fully volumetric neural disentanglement framework for source appearance and driver expressions. Our method is real-time and produces high-fidelity and view-consistent output, suitable for 3D teleconferencing systems based on holographic displays. Existing cutting-edge 3D-aware reenactment methods often use neural radiance fields or 3D meshes to produce view-consistent appearance encoding, but, at the same time, they rely on linear face models, such as 3DMM, to achieve its disentanglement with facial expressions. As a result, their reenactment results often exhibit identity leakage from the driver or have unnatural expressions. To address these problems, we propose a neural self-supervised disentanglement approach that lifts both the source image and driver video frame into a shared 3D volumetric representation based on tri-planes. This representation can then be freely manipulated with expression tri-planes extracted from the driving images and rendered from an arbitrary view using neural radiance fields. We achieve this disentanglement via self-supervised learning on a large in-the-wild video dataset. We further introduce a highly effective fine-tuning approach to improve the generalizability of the 3D lifting using the same real-world data. We demonstrate state-of-the-art performance on a wide range of datasets, and also showcase high-quality 3D-aware head reenactment on highly challenging and diverse subjects, including non-frontal head poses and complex expressions for both source and driver.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
