SUR-adapter：利用大语言模型增强文本到图像预训练扩散模型
SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models
AI总结：
本文提出 SUR-adapter，通过构建 SURD 数据集并利用大语言模型知识蒸馏，增强扩散模型对简洁叙述提示的语义理解与推理能力，从而提升文本到图像生成质量。
摘要：
扩散模型已成为流行的文本到图像生成模型，能够在文本提示的引导下生成高质量、内容丰富的图像。然而，当输入提示为简洁叙述时，现有模型在语义理解和常识推理方面存在局限，导致图像生成质量低下。为提升模型对叙述性提示的处理能力，我们提出了一种简单而有效的参数高效微调方法，称为语义理解与推理适配器（SUR-adapter），用于预训练扩散模型。为实现这一目标，我们首先收集并标注了一个新数据集 SURD，该数据集包含超过 57,000 个经过语义校正的多模态样本。每个样本包含一个简洁叙述提示、一个基于复杂关键词的提示以及一张高质量图像。随后，我们将叙述提示的语义表示与复杂提示对齐，并通过知识蒸馏将大语言模型（LLMs）的知识迁移到我们的 SUR-adapter 中，使其能够获得强大的语义理解和推理能力，从而为文本到图像生成构建高质量的文本语义表示。我们通过集成多个大语言模型和流行的预训练扩散模型进行实验，证明了我们的方法在使扩散模型能够理解和推理简洁自然语言的同时不降低图像质量方面的有效性。我们的方法可以使文本到图像扩散模型更易于使用，带来更好的用户体验，这表明我们的方法有潜力通过弥合简洁叙述提示与基于复杂关键词的提示之间的语义鸿沟，进一步推动用户友好型文本到图像生成模型的发展。代码已发布在 https://github.com/Qrange-group/SUR-adapter。
英文摘要：
Diffusion models, which have emerged to become popular text-to-image generation models, can produce high-quality and content-rich images guided by textual prompts. However, there are limitations to semantic understanding and commonsense reasoning in existing models when the input prompts are concise narrative, resulting in low-quality image generation. To improve the capacities for narrative prompts, we propose a simple-yet-effective parameter-efficient fine-tuning approach called the Semantic Understanding and Reasoning adapter (SUR-adapter) for pre-trained diffusion models. To reach this goal, we first collect and annotate a new dataset SURD which consists of more than 57,000 semantically corrected multi-modal samples. Each sample contains a simple narrative prompt, a complex keyword-based prompt, and a high-quality image. Then, we align the semantic representation of narrative prompts to the complex prompts and transfer knowledge of large language models (LLMs) to our SUR-adapter via knowledge distillation so that it can acquire the powerful semantic understanding and reasoning capabilities to build a high-quality textual semantic representation for text-to-image generation. We conduct experiments by integrating multiple LLMs and popular pre-trained diffusion models to show the effectiveness of our approach in enabling diffusion models to understand and reason concise natural language without image quality degradation. Our approach can make text-to-image diffusion models easier to use with better user experience, which demonstrates our approach has the potential for further advancing the development of user-friendly text-to-image generation models by bridging the semantic gap between simple narrative prompts and complex keyword-based prompts. The code is released at https://github.com/Qrange-group/SUR-adapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
