arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.08453cs.CVcs.AIcs.LGcs.MM
收藏

T2I-Adapter:挖掘文本到图像扩散模型更可控能力的学习适配器

T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);University of Macau(澳门大学);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Peking University Shenzhen Graduate School(北京大学深圳研究生院)

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie

摘要:

大规模文本到图像(T2I)模型令人难以置信的生成能力已经证明了学习复杂结构和有意义语义的强大能力。然而,仅仅依靠文本提示不能充分利用模型学习的知识,特别是在灵活和准确的控制(例如,颜色和结构)。在本文中,我们的目标是“挖掘”T2I模型已经隐含学习的能力,然后显式地使用它们来更细粒度地控制生成。具体来说,我们建议学习简单而轻量级的T2I适配器,以将T2I模型中的内部知识与外部控制信号相匹配,同时冻结原始的大型T2I模型。这样就可以根据不同的条件训练各种适配器,在生成结果的颜色和结构上实现丰富的控制和编辑效果。此外,所提出的T2I适配器具有吸引人的实用价值,如可组合性和泛化能力。大量的实验表明,我们的T2I适配器具有良好的生成质量和广泛的应用。

英文摘要:

The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., color and structure) is needed. In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly. Specifically, we propose to learn simple and lightweight T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models. In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results. Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability. Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑