CLIPAG：迈向无生成器的文本到图像生成
CLIPAG: Towards Generator-Free Text-to-Image Generation
AI总结：
本研究将感知对齐梯度扩展到视觉-语言模型，通过对抗性微调CLIP获得CLIPAG，实现即插即用改进生成任务，并无需生成器即可完成文本到图像生成。
摘要：
感知对齐梯度（PAG）是指在鲁棒图像分类模型中观察到的一个引人入胜的特性，其中模型的输入梯度与人类感知对齐并具有语义含义。尽管这一现象已引起广泛研究关注，但先前仅在单模态纯视觉架构的背景下进行了研究。在本工作中，我们将PAG的研究扩展到视觉-语言架构，这些架构构成了多种图像-文本任务和应用的基础。通过对CLIP进行对抗性鲁棒化微调，我们证明了鲁棒的视觉-语言模型表现出PAG，而它们的普通对应模型则不然。这项工作揭示了具有PAG的CLIP（CLIPAG）在若干视觉-语言生成任务中的优势。值得注意的是，我们表明，以“即插即用”的方式无缝集成CLIPAG，能够显著改进视觉-语言生成应用。此外，利用其PAG特性，CLIPAG使得无需任何生成模型即可实现文本到图像生成，而通常这需要庞大的生成器。
英文摘要：
Perceptually Aligned Gradients (PAG) refer to an intriguing property observed in robust image classification models, wherein their input gradients align with human perception and pose semantic meanings. While this phenomenon has gained significant research attention, it was solely studied in the context of unimodal vision-only architectures. In this work, we extend the study of PAG to Vision-Language architectures, which form the foundations for diverse image-text tasks and applications. Through an adversarial robustification finetuning of CLIP, we demonstrate that robust Vision-Language models exhibit PAG in contrast to their vanilla counterparts. This work reveals the merits of CLIP with PAG (CLIPAG) in several vision-language generative tasks. Notably, we show that seamlessly integrating CLIPAG in a "plug-n-play" manner leads to substantial improvements in vision-language generative applications. Furthermore, leveraging its PAG property, CLIPAG enables text-to-image generation without any generative model, which typically requires huge generators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
