基于 Diffusion Prior 的受控与条件文本到图像生成
Controlled and Conditional Text to Image Generation with Diffusion Prior
AI总结：
本文探索 DALLE-2 的 Diffusion Prior 能力，提出利用其以高效内存和计算方式约束生成至特定领域，并通过颜色直方图等条件信息进一步控制生成，定量与定性实验证明该方法优于提示工程和现有基线。
摘要：
去噪扩散模型在从文本生成多样化、高质量图像方面表现出卓越性能。在 Stable Diffusion 和 Imagen 等直接从文本生成图像的模型基础上或与之对齐，已提出了大量技术。较少探索的方法是 DALLE-2 的两步过程，包括从文本生成 CLIP 图像嵌入的 Diffusion Prior 和从 CLIP 图像嵌入生成图像的 Diffusion Decoder。我们探索了 Diffusion Prior 的能力及中间 CLIP 表示的优势。我们观察到，Diffusion Prior 能以内存和计算高效的方式将生成约束到特定领域，而无需改变更大的 Diffusion Decoder。此外，我们表明 Diffusion Prior 可通过额外条件信息（如颜色直方图）训练，以进一步控制生成。我们定量和定性地表明，所提方法在特定领域生成上优于提示工程，在颜色条件生成上优于现有基线。我们相信，这些观察与结果将推动对扩散先验的进一步研究，揭示其更多能力。
英文摘要：
Denoising Diffusion models have shown remarkable performance in generating diverse, high quality images from text. Numerous techniques have been proposed on top of or in alignment with models like Stable Diffusion and Imagen that generate images directly from text. A lesser explored approach is DALLE-2's two step process comprising a Diffusion Prior that generates a CLIP image embedding from text and a Diffusion Decoder that generates an image from a CLIP image embedding. We explore the capabilities of the Diffusion Prior and the advantages of an intermediate CLIP representation. We observe that Diffusion Prior can be used in a memory and compute efficient way to constrain the generation to a specific domain without altering the larger Diffusion Decoder. Moreover, we show that the Diffusion Prior can be trained with additional conditional information such as color histogram to further control the generation. We show quantitatively and qualitatively that the proposed approaches perform better than prompt engineering for domain specific generation and existing baselines for color conditioned generation. We believe that our observations and results will instigate further research into the diffusion prior and uncover more of its capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
