用于单图像高保真3D头部重建的双编码器GAN反演
Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images
AI总结：
本文提出基于PanoHead的双编码器GAN反演框架，通过三平面域拼接和遮挡感知判别器，实现单图像高保真3D头部重建与多视角生成。
摘要：
3D GAN反演旨在将单张图像投影到3D生成对抗网络（GAN）的潜在空间中，从而实现3D几何重建。尽管已有编码器在3D GAN反演中取得了良好效果，但它们主要构建在EG3D之上，而EG3D擅长合成近正面视图，在从多样视角合成全面3D场景方面存在局限。与现有方法不同，我们提出了一个构建在PanoHead之上的新型框架，PanoHead擅长从360度视角合成图像。为了实现输入图像的真实感3D建模，我们引入了一个双编码器系统，专门用于高保真重建和从不同视角进行真实感生成。与此同时，我们提出了一个在三平面域上的拼接框架，以从两者中获得最佳预测。为了实现无缝拼接，尽管两个编码器专精于不同任务，它们仍必须输出一致的结果。为此，我们使用专门的损失函数精心训练这些编码器，其中包括基于我们新颖的遮挡感知三平面判别器的对抗损失。实验表明，我们的方法在定性和定量上都超越了现有的编码器训练方法。请访问项目页面：https://berkegokmen1.github.io/dual-enc-3d-gan-inv。
英文摘要：
3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results in 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and is limiting in synthesizing comprehensive 3D scenes from diverse viewpoints. In contrast to existing approaches, we propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective. To achieve realistic 3D modeling of the input image, we introduce a dual encoder system tailored for high-fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, we propose a stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively. Please visit the project page: https://berkegokmen1.github.io/dual-enc-3d-gan-inv.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
