思考-然后-生成：基于LLM编码器的推理感知文本到图像扩散模型
Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders
AI总结：
本文提出基于LLM编码器的推理感知文本到图像扩散模型，通过推理重写提示提升生成质量，达到接近GPT-4的性能。
摘要：
近期文本到图像（T2I）扩散模型（DMs）的进步使从多样文本提示生成高质量视觉合成成为可能。然而，大多数现有的T2I DMs，即使那些配备了基于大语言模型（LLM）的文本编码器，仍然只是文本-像素映射器——它们仅将LLMs用作文本编码器，而没有利用其固有的推理能力来推断给定文本提示应视觉上表现什么。为了超越这种字面生成，我们提出了思考-然后-生成（T2G）范式，其中基于LLM的文本编码器被鼓励对原始用户提示进行推理并重写；重写后的提示状态则作为扩散条件。为此，我们首先通过轻量级的监督微调过程激活LLM编码器的思考-然后-重写模式。随后，LLM编码器和扩散主干共同优化，以确保通过Dual-GRPO实现对上下文的忠实推理和语义的准确渲染。特别是，文本编码器通过图像接地奖励来推断和回忆世界知识，而扩散主干则被推动生成语义一致且视觉连贯的图像。实验显示，在基于推理的图像生成和编辑基准中，事实一致性、语义对齐和视觉真实感均有所提升，达到0.79的WISE分数，接近GPT-4的水平。我们的结果构成了朝着具有推理、表达和演示能力的下一代统一模型迈出的积极一步。
英文摘要：
Recent progress in text-to-image (T2I) diffusion models (DMs) has enabled high-quality visual synthesis from diverse textual prompts. Yet, most existing T2I DMs, even those equipped with large language model (LLM)-based text encoders, remain text-pixel mappers -- they employ LLMs merely as text encoders, without leveraging their inherent reasoning capabilities to infer what should be visually depicted given the textual prompt. To move beyond such literal generation, we propose the think-then-generate (T2G) paradigm, where the LLM-based text encoder is encouraged to reason about and rewrite raw user prompts; the states of the rewritten prompts then serve as diffusion conditioning. To achieve this, we first activate the think-then-rewrite pattern of the LLM encoder with a lightweight supervised fine-tuning process. Subsequently, the LLM encoder and diffusion backbone are co-optimized to ensure faithful reasoning about the context and accurate rendering of the semantics via Dual-GRPO. In particular, the text encoder is reinforced using image-grounded rewards to infer and recall world knowledge, while the diffusion backbone is pushed to produce semantically consistent and visually coherent images. Experiments show substantial improvements in factual consistency, semantic alignment, and visual realism across reasoning-based image generation and editing benchmarks, achieving 0.79 on WISE score, nearly on par with GPT-4. Our results constitute a promising step toward next-generation unified models with reasoning, expression, and demonstration capacities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
