当StyleGAN遇上Stable Diffusion：一种用于个性化图像生成的$\mathscr{W}_+$适配器
When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation
AI总结：
本文提出将StyleGAN的扩展嵌入空间$\mathcal{W}_+$与文本到图像扩散模型对齐，通过新的训练目标平衡提示与身份条件，实现身份保持与语义编辑兼顾的个性化图像生成。
摘要：
文本到图像扩散模型在生成多样化、高质量和照片级真实感图像方面取得了显著进展。这一进步激发了人们将特定身份融入生成内容的日益浓厚的兴趣。目前大多数方法采用反演方法，利用单张参考图像将目标视觉概念嵌入到文本嵌入空间中。然而，新合成的人脸要么在面部属性（如表情）上与参考图像高度相似，要么身份保持能力下降。由于身份信息与从参考图像中提取的与身份无关的面部属性之间存在复杂的纠缠，旨在引导合成人脸面部属性的文本描述可能效果不佳。为了解决这些问题，我们提出了对扩展StyleGAN嵌入空间$\mathcal{W}_+$的新颖应用，以实现扩散模型更强的身份保持和属性解耦。通过将这一具有语义意义的人脸潜在空间与文本到图像扩散模型对齐，我们成功地在保持身份高保真度的同时，具备语义编辑能力。此外，我们提出了新的训练目标来平衡提示条件和身份条件的影响，确保在修改面部属性时与身份无关的背景不受影响。大量实验表明，我们的方法能够熟练地生成个性化文本到图像输出，这些输出不仅与提示描述兼容，而且在多种设置下也适用于常见的StyleGAN编辑方向。我们的源代码将在\url{https://github.com/csxmli2016/w-plus-adapter}上公开。
英文摘要：
Text-to-image diffusion models have remarkably excelled in producing diverse, high-quality, and photo-realistic images. This advancement has spurred a growing interest in incorporating specific identities into generated content. Most current methods employ an inversion approach to embed a target visual concept into the text embedding space using a single reference image. However, the newly synthesized faces either closely resemble the reference image in terms of facial attributes, such as expression, or exhibit a reduced capacity for identity preservation. Text descriptions intended to guide the facial attributes of the synthesized face may fall short, owing to the intricate entanglement of identity information with identity-irrelevant facial attributes derived from the reference image. To address these issues, we present the novel use of the extended StyleGAN embedding space $\mathcal{W}_+$, to achieve enhanced identity preservation and disentanglement for diffusion models. By aligning this semantically meaningful human face latent space with text-to-image diffusion models, we succeed in maintaining high fidelity in identity preservation, coupled with the capacity for semantic editing. Additionally, we propose new training objectives to balance the influences of both prompt and identity conditions, ensuring that the identity-irrelevant background remains unaffected during facial attribute modifications. Extensive experiments reveal that our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions in diverse settings. Our source code will be available at \url{https://github.com/csxmli2016/w-plus-adapter}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
