arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.04279cs.CV
收藏

基于文本到图像扩散模型的可控生成:综述

Controllable Generation with Text-to-Image Diffusion Models: A Survey

Beijing University of Posts and Telecommunications(北京邮电大学)

Pu Cao, Feng Zhou, Qing Song, Lu Yang

AI总结:

本文综述了基于文本到图像扩散模型的可控生成方法,分析了不同条件下的生成机制及分类。

摘要:

在视觉生成领域迅速发展的背景下,扩散模型彻底改变了生成图像的面貌,其出色的文本引导生成功能标志着能力上的显著飞跃。然而,仅依赖文本对这些模型进行条件化并不能完全满足不同应用和场景中多样化和复杂的需求。鉴于这一不足,各种研究旨在控制预训练的文本到图像(T2I)模型以支持新的条件。在本次综述中,我们对基于T2I扩散模型的可控生成文献进行了全面回顾,涵盖了该领域的理论基础和实际进展。我们的回顾从简要介绍去噪扩散概率模型(DDPMs)和广泛使用的T2I扩散模型开始。然后揭示扩散模型的控制机制,从理论上分析如何将新条件引入去噪过程以实现条件生成。此外,我们还提供了该领域研究的详细概述,从条件角度将其分为不同的类别:特定条件下的生成、多条件下的生成以及通用可控生成。如需查看所调查的可控生成文献的完整列表,请参考我们整理的仓库:https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models。

英文摘要:

In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑