arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.07536cs.CV
收藏

FreeControl:对任意文本到图像扩散模型实现带任意条件的免训练空间控制

FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition

University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Innopeak Technology, Inc(Innopeak科技公司)

Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, Bolei Zhou

AI总结:

针对现有文本到图像扩散模型空间控制方法需为不同条件、架构和检查点单独训练辅助模块的问题,提出免训练方法FreeControl,通过结构与外观引导实现多条件适配,性能媲美有训练方法。

摘要:

ControlNet等近期方法为用户提供了对文本到图像(T2I)扩散模型的细粒度空间控制能力。但这类方法必须为每种空间条件、模型架构和检查点单独训练辅助模块,与人类设计师在内容创作过程中想向AI模型传递的多样化意图与偏好相矛盾。本研究提出FreeControl,这是一种免训练的可控T2I生成方法,可同时支持多种条件、架构与检查点。FreeControl设计了结构引导机制以实现与引导图像的结构对齐,还设计了外观引导机制,让使用同一随机种子生成的图像之间可以共享外观。大量定性与定量实验表明,FreeControl在多种预训练T2I模型上均表现出优异性能。具体而言,FreeControl可便捷地对多种不同架构和检查点实现免训练控制,能处理多数现有免训练方法无法应对的高难度输入条件,且合成质量与基于训练的方法相当。

英文摘要:

Recent approaches such as ControlNet offer users fine-grained spatial control over text-to-image (T2I) diffusion models. However, auxiliary modules have to be trained for each type of spatial condition, model architecture, and checkpoint, putting them at odds with the diverse intents and preferences a human designer would like to convey to the AI models during the content creation process. In this work, we present FreeControl, a training-free approach for controllable T2I generation that supports multiple conditions, architectures, and checkpoints simultaneously. FreeControl designs structure guidance to facilitate the structure alignment with a guidance image, and appearance guidance to enable the appearance sharing between images generated using the same seed. Extensive qualitative and quantitative experiments demonstrate the superior performance of FreeControl across a variety of pre-trained T2I models. In particular, FreeControl facilitates convenient training-free control over many different architectures and checkpoints, allows the challenging input conditions on which most of the existing training-free methods fail, and achieves competitive synthesis quality with training-based approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑