Magnet:在理解视觉-语言模型如何运作之前,我们永远无法知晓文本到图像扩散模型的工作机制
Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
AI总结:
本文提出无需训练的 Magnet 方法,通过正负绑定向量和邻居策略解决文本到图像扩散模型中的属性绑定问题,显著提升复杂提示下的图像合成质量。
摘要:
文本到图像扩散模型,尤其是 Stable Diffusion,已经彻底改变了计算机视觉领域。然而,当要求生成能够忠实表示涉及多个属性和对象的复杂提示的图像时,合成质量往往会下降。以往研究表明,混合文本嵌入会导致属性绑定不当,但很少有工作对此进行深入探索。在这项工作中,我们批判性地审视了 CLIP 文本编码器在理解属性方面的局限性,并研究了这如何影响扩散模型。我们识别出文本空间中的属性偏差现象,并强调了填充嵌入中纠缠不同概念的上下文问题。我们提出了 Magnet,一种无需训练的新方法来解决属性绑定问题。我们引入正绑定向量和负绑定向量来增强解纠缠,并进一步采用邻居策略来提高准确性。大量实验表明,Magnet 以可忽略的计算成本显著提高了合成质量和绑定准确性,使得生成非常规和非自然概念成为可能。
英文摘要:
Text-to-image diffusion models particularly Stable Diffusion, have revolutionized the field of computer vision. However, the synthesis quality often deteriorates when asked to generate images that faithfully represent complex prompts involving multiple attributes and objects. While previous studies suggest that blended text embeddings lead to improper attribute binding, few have explored this in depth. In this work, we critically examine the limitations of the CLIP text encoder in understanding attributes and investigate how this affects diffusion models. We discern a phenomenon of attribute bias in the text space and highlight a contextual issue in padding embeddings that entangle different concepts. We propose \textbf{Magnet}, a novel training-free approach to tackle the attribute binding problem. We introduce positive and negative binding vectors to enhance disentanglement, further with a neighbor strategy to increase accuracy. Extensive experiments show that Magnet significantly improves synthesis quality and binding accuracy with negligible computational cost, enabling the generation of unconventional and unnatural concepts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。