改进文本到图像生成模型中的组合属性绑定：通过增强文本嵌入
Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings
AI总结：
本文针对文本到图像模型组合属性绑定失败问题，发现CLIP文本编码器输出空间次优且最终token嵌入含无关注意力，通过微调参数高效线性投影，在不损害FID分数下显著提升组合生成质量。
摘要：
基于扩散的文本到图像生成模型具有生成照片级逼真图像的惊人能力，并在具有挑战性的图像生成基准上取得了最先进的低FID分数。然而，这些文本到图像生成模型的主要失败模式之一在于将属性、对象及其关联关系准确地组合到图像中。在我们的论文中，我们研究了组合属性绑定失败，即模型未能将描述性属性（如颜色、形状或纹理）与生成图像中的相应对象正确关联，并指出使用CLIP文本编码器的不完美文本条件化是这些模型无法生成高保真组合场景的主要原因之一。特别地，我们表明：（i）存在一个最优的文本嵌入空间，可以生成高度连贯的组合场景，这表明CLIP文本编码器的输出空间是次优的；（ii）CLIP中的最终token嵌入是错误的，因为它们通常包含来自组合提示中无关token的注意力贡献。我们的主要发现表明，通过在Stable-Diffusion变体中对CLIP的表示空间仅微调一个简单且参数高效的线性投影，并使用一小组合成图像-文本对，可以实现显著的组合改进（而不会损害模型的FID分数）。
英文摘要：
Text-to-image diffusion-based generative models have the stunning ability to generate photo-realistic images and achieve state-of-the-art low FID scores on challenging image generation benchmarks. However, one of the primary failure modes of these text-to-image generative models is in composing attributes, objects, and their associated relationships accurately into an image. In our paper, we investigate compositional attribute binding failures, where the model fails to correctly associate descriptive attributes (such as color, shape, or texture) with the corresponding objects in the generated images, and highlight that imperfect text conditioning with CLIP text-encoder is one of the primary reasons behind the inability of these models to generate high-fidelity compositional scenes. In particular, we show that (i) there exists an optimal text-embedding space that can generate highly coherent compositional scenes showing that the output space of the CLIP text-encoder is sub-optimal, and (ii) the final token embeddings in CLIP are erroneous as they often include attention contributions from unrelated tokens in compositional prompts. Our main finding shows that significant compositional improvements can be achieved (without harming the model's FID score) by fine-tuning only a simple and parameter-efficient linear projection on CLIP's representation space in Stable-Diffusion variants using a small set of compositional image-text pairs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
