用单手草图定制 StyleGAN
Customize StyleGAN with One Hand Sketch
AI总结：
本文提出一种基于能量学习和CLIP跨域语义监督的框架，仅用单个草图即可控制预训练StyleGAN生成语义对齐的多模态图像，在无额外训练数据的严格设置下优于先前方法。
摘要：
从人类草图生成图像通常需要从头训练专用网络。相比之下，预训练视觉-语言模型（如 CLIP）的出现推动了基于文本输入或参考图像控制现有 StyleGAN 模型输出图像的生成应用。与此同时，我们的工作提出了一个框架，用单个用户草图控制 StyleGAN 图像。具体来说，我们通过基于能量的学习在预训练 StyleGAN 模型的潜在空间中学习条件分布，并提出两个利用 CLIP 进行跨域语义监督的新颖能量函数。训练完成后，我们的模型可以生成与输入草图语义对齐的多模态图像。在合成数据集上的定量评估表明，我们的方法在单次（one-shot）场景下相比先前方法有显著改进。当在风格和姿态多样的广泛人类草图上进行实验时，我们方法的优越性进一步得到凸显。令人惊讶的是，尽管在更严格的设置下运行——没有额外训练数据且仅输入单个草图——我们的模型在草图输入范围和图像质量两方面都优于先前的基线。
英文摘要：
Generating images from human sketches typically requires dedicated networks trained from scratch. In contrast, the emergence of the pre-trained Vision-Language models (e.g., CLIP) has propelled generative applications based on controlling the output imagery of existing StyleGAN models with text inputs or reference images. Parallelly, our work proposes a framework to control StyleGAN imagery with a single user sketch. In particular, we learn a conditional distribution in the latent space of a pre-trained StyleGAN model via energy-based learning and propose two novel energy functions leveraging CLIP for cross-domain semantic supervision. Once trained, our model can generate multi-modal images semantically aligned with the input sketch. Quantitative evaluations on synthesized datasets have shown that our approach improves significantly from previous methods in the one-shot regime. The superiority of our method is further underscored when experimenting with a wide range of human sketches of diverse styles and poses. Surprisingly, our models outperform the previous baseline regarding both the range of sketch inputs and image qualities despite operating with a stricter setting: with no extra training data and single sketch input.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
