arXiv 2306.046362023-06-07cs.CVcs.LG

GP-UNIT：面向通用无监督图像到图像翻译的生成先验

GP-UNIT: Generative Prior for Versatile Unsupervised Image-to-Image Translation

Nanyang Technological University（南洋理工大学）

Shuai Yang, Liming Jiang, Ziwei Liu, Chen Change Loy

备注：: Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). Code: https://github.com/williamyang1991/GP-UNIT Project page: https://www.mmlab-ntu.com/project/gpunit/. arXiv admin note: substantial text overlap with arXiv:2204.03641

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对无监督图像到图像翻译中跨域映射鲁棒性不足的问题，提出GP-UNIT框架，通过预训练类别条件GAN的生成先验构建多层级跨域对应，实现远近域高质量可控翻译，性能优于现有最优模型。

摘要：

深度学习的最新进展催生了许多成功的无监督图像到图像翻译模型，这类模型无需配对数据即可学习两个视觉域之间的对应关系。然而，在不同域之间构建鲁棒的映射仍是巨大挑战，尤其是视觉差异极大的域之间。本文提出一种新型通用框架——生成先验引导的无监督图像到图像翻译（Generative Prior-guided UNsupervised Image-to-image Translation，GP-UNIT），可提升现有翻译模型的质量、适用性与可控性。GP-UNIT的核心思路是从预训练的类别条件GAN中提取生成先验，构建粗粒度跨域对应关系，再将学习到的先验应用于对抗翻译以挖掘细粒度对应关系。借助学习到的多层级内容对应关系，GP-UNIT能够在相近域和远域之间实现有效翻译。对于相近域，GP-UNIT可基于一个参数控制翻译过程中内容对应关系的强度，让用户在内容一致性与风格一致性之间取得平衡。对于远域，该研究探索了半监督学习方法，引导GP-UNIT发现仅靠外观难以学习的准确语义对应关系。我们通过大量实验验证了GP-UNIT在各类域之间实现鲁棒、高质量、多样化翻译方面优于现有最优翻译模型。

英文摘要：

Recent advances in deep learning have witnessed many successful unsupervised image-to-image translation models that learn correspondences between two visual domains without paired data. However, it is still a great challenge to build robust mappings between various domains especially for those with drastic visual discrepancies. In this paper, we introduce a novel versatile framework, Generative Prior-guided UNsupervised Image-to-image Translation (GP-UNIT), that improves the quality, applicability and controllability of the existing translation models. The key idea of GP-UNIT is to distill the generative prior from pre-trained class-conditional GANs to build coarse-level cross-domain correspondences, and to apply the learned prior to adversarial translations to excavate fine-level correspondences. With the learned multi-level content correspondences, GP-UNIT is able to perform valid translations between both close domains and distant domains. For close domains, GP-UNIT can be conditioned on a parameter to determine the intensity of the content correspondences during translation, allowing users to balance between content and style consistency. For distant domains, semi-supervised learning is explored to guide GP-UNIT to discover accurate semantic correspondences that are hard to learn solely from the appearance. We validate the superiority of GP-UNIT over state-of-the-art translation models in robust, high-quality and diversified translations between various domains through extensive experiments.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。