Rare-to-Frequent：利用 LLM 引导释放扩散模型在罕见概念上的组合生成能力
Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance
AI总结：
针对扩散模型难以生成罕见概念组合的问题，本文提出无需训练的 R2F，利用 LLM 在推理中进行罕见到频繁概念引导，并在 RareBench 等数据集上显著提升 T2I 对齐度。
摘要：
最先进的文本到图像（T2I）扩散模型往往难以生成概念的罕见组合，例如具有不寻常属性的对象。本文表明，通过大语言模型（LLM）引导，可以显著增强扩散模型在此类罕见概念上的组合生成能力。我们首先进行实证和理论分析，证明在扩散采样过程中呈现与目标罕见概念相关的频繁概念，能够产生更准确的概念组合。基于此，我们提出了一种无需训练的方法 R2F，该方法利用 LLM 中丰富的语义知识，在整个扩散推理过程中规划并执行从罕见概念到频繁概念的整体引导。我们的框架可灵活适用于任意预训练扩散模型和 LLM，并且能够与区域引导扩散方法无缝集成。在三个数据集上的大量实验（包括我们新提出的基准 RareBench，其中包含各种具有罕见概念组合的提示词）表明，R2F 在 T2I 对齐度上显著优于包括 SD3.0 和 FLUX 在内的现有模型，提升幅度最高达 28.1 个百分点。代码可在 https://github.com/krafton-ai/Rare-to-Frequent 获取。
英文摘要：
State-of-the-art text-to-image (T2I) diffusion models often struggle to generate rare compositions of concepts, e.g., objects with unusual attributes. In this paper, we show that the compositional generation power of diffusion models on such rare concepts can be significantly enhanced by the Large Language Model (LLM) guidance. We start with empirical and theoretical analysis, demonstrating that exposing frequent concepts relevant to the target rare concepts during the diffusion sampling process yields more accurate concept composition. Based on this, we propose a training-free approach, R2F, that plans and executes the overall rare-to-frequent concept guidance throughout the diffusion inference by leveraging the abundant semantic knowledge in LLMs. Our framework is flexible across any pre-trained diffusion models and LLMs, and can be seamlessly integrated with the region-guided diffusion approaches. Extensive experiments on three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly surpasses existing models including SD3.0 and FLUX by up to 28.1%p in T2I alignment. Code is available at https://github.com/krafton-ai/Rare-to-Frequent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
