arXiv 2310.142222024-12-07cs.CV

One-for-All：面向使用单个StyleGAN的通用域翻译

One-for-All: Towards Universal Domain Translation with a Single StyleGAN

School of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；Shanghai Jiao Tong University（上海交通大学）；School of Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学学院）；University of California at Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google（谷歌公司）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）

Yong Du, Jiahui Zhan, Xinzhe Li, Junyu Dong, Sheng Chen, Ming-Hsuan Yang, Shengfeng He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对训练数据有限、域间视觉差异大的跨域翻译问题，提出融合CLIP域中立语义与目标语义、搭配CLIP2P映射器对接StyleGAN的UniTranslator模型，通用且效果优于现有通用模型、可比肩专用模型。

摘要：

本文提出了一种新型翻译模型UniTranslator，用于在训练数据有限且视觉差异显著的条件下，在视觉差异明显的域之间转换表征。我们方法的核心思路是利用CLIP（对比语言-图像预训练）的域中立能力作为桥接机制，同时使用一个独立模块从源域和目标域的嵌入中提取抽象的、与域无关的语义。将这些抽象语义与目标特定语义相融合，可得到CLIP空间内的转换后嵌入。为弥合CLIP与StyleGAN两个不同体系之间的差距，我们提出了一种新型非线性映射器CLIP2P mapper。该模块利用CLIP嵌入，专门用于近似StyleGAN潜在空间中的潜在分布，有效充当这两个空间之间的连接器。所提出的UniTranslator具有通用性，能够执行多种任务，包括风格混合、风格化以及域翻译，即便在跨不同视觉域的视觉挑战性场景中也能适用。值得注意的是，UniTranslator生成的高质量翻译结果具备域相关性、多样性，且图像质量有所提升。UniTranslator的性能超越了现有通用模型，在代表性任务中与专用模型相比表现出色。源代码和训练后的模型将向公众发布。

英文摘要：

In this paper, we propose a novel translation model, UniTranslator, for transforming representations between visually distinct domains under conditions of limited training data and significant visual differences. The main idea behind our approach is leveraging the domain-neutral capabilities of CLIP as a bridging mechanism, while utilizing a separate module to extract abstract, domain-agnostic semantics from the embeddings of both the source and target realms. Fusing these abstract semantics with target-specific semantics results in a transformed embedding within the CLIP space. To bridge the gap between the disparate worlds of CLIP and StyleGAN, we introduce a new non-linear mapper, the CLIP2P mapper. Utilizing CLIP embeddings, this module is tailored to approximate the latent distribution in the StyleGAN's latent space, effectively acting as a connector between these two spaces. The proposed UniTranslator is versatile and capable of performing various tasks, including style mixing, stylization, and translations, even in visually challenging scenarios across different visual domains. Notably, UniTranslator generates high-quality translations that showcase domain relevance, diversity, and improved image quality. UniTranslator surpasses the performance of existing general-purpose models and performs well against specialized models in representative tasks. The source code and trained models will be released to the public.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。