JoIN: 用于内在图像分解的联合GAN反向求解
JoIN: Joint GANs Inversion for Intrinsic Image Decomposition
AI总结：
JoIN通过联合反向求解多个GAN实现内在图像分解，利用合成数据训练GAN先验并微调生成器，以提高真实图像的泛化能力。
摘要：
内在图像分解（IID）是一个具有挑战性的逆向问题，旨在将自然图像分解为其内在组成部分，如反光率和阴影。尽管最近的图像分解方法依赖于这些组件的学习先验知识，但它们往往因联合训练先验知识而遭受组件交叉污染；或因先验知识在真实图像上的推理过程中保持冻结而产生真实与合成之间的差距。在本工作中，我们提出使用生成对抗网络（GANs）的银行作为先验知识来解决内在图像分解问题，其中每个GAN仅独立地在单一内在组件上训练，提供更强且更解耦的先验知识。我们方法的核心思想是GAN的潜在空间是一个适合解决逆向问题的优化领域。给定一个输入图像，我们提出联合反向求解一组GANs的潜在代码并结合其输出以重现输入。与现有所有GAN反向求解方法仅限于反向求解单个GAN不同，我们提出的JoIN方法能够仅使用单张图像作为监督，同时保持每个内在组件的分布先验。我们证明了我们的方法是模块化的，允许各种正向成像模型，并且能够成功分解合成和真实图像。进一步地，受现有GAN反向求解方法的启发，我们在真实图像上的推理过程中允许对生成器先验进行细致的微调。这样，尽管仅使用合成数据训练GAN先验，我们的方法仍能在真实图像上实现优异的泛化能力。我们通过各种数据集上的详尽定性和定量评估和消融研究证明了我们方法的成功。
英文摘要：
Intrinsic Image Decomposition (IID) is a challenging inverse problem that seeks to decompose a natural image into its underlying intrinsic components such as albedo and shading. While recent image decomposition methods rely on learning-based priors on these components, they often suffer from component cross-contamination owing to joint training of priors; or from Sim-to-Real gap since the priors trained on synthetic data are kept frozen during the inference on real images. In this work, we propose to solve the intrinsic image decomposition problem using a bank of Generative Adversarial Networks (GANs) as priors where each GAN is independently trained only on a single intrinsic component, providing stronger and more disentangled priors. At the core of our approach is the idea that the latent space of a GAN is a well-suited optimization domain to solve inverse problems. Given an input image, we propose to jointly invert the latent codes of a set of GANs and combine their outputs to reproduce the input. Contrary to all existing GAN inversion methods that are limited to inverting only a single GAN, our proposed approach, JoIN, is able to jointly invert multiple GANs using only a single image as supervision while still maintaining distribution priors of each intrinsic component. We show that our approach is modular, allowing various forward imaging models, and that it can successfully decompose both synthetic and real images. Further, taking inspiration from existing GAN inversion approaches, we allow for careful fine-tuning of the generator priors during the inference on real images. This way, our method is able to achieve excellent generalization on real images even though it uses only synthetic data to train the GAN priors. We demonstrate the success of our approach through exhaustive qualitative and quantitative evaluations and ablation studies on various datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
