arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07540cs.CVcs.LG
收藏

Ctrl-X:无引导下控制文本到图像生成的结构与外观

Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance

University of California, Los Angeles(加州大学洛杉矶分校);NVIDIA(英伟达公司)

Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, Bolei Zhou

AI总结:

针对现有无辅助模块的T2I可控生成方法步骤多、耗时长的问题,提出Ctrl-X框架,通过前馈结构控制和语义感知外观迁移实现无引导控制,性能更优且支持即插即用。

摘要:

近期的可控生成方法,如FreeControl(自由控制)和Diffusion Self-Guidance(扩散自引导),无需训练辅助模块,就能为文本到图像(T2I)扩散模型带来细粒度的空间与外观控制。然而,这些方法针对每一类得分函数优化潜在嵌入时,需要更多扩散步骤,导致生成过程耗时较长,限制了其灵活性与应用场景。本研究提出Ctrl-X,这是一个无需额外训练或引导、用于T2I扩散模型结构与外观控制的简易框架。Ctrl-X设计了前馈结构控制模块,实现与结构图像的结构对齐;还设计了语义感知外观迁移模块,助力从用户输入图像迁移外观。大量定性与定量实验表明,Ctrl-X在多种条件输入和模型检查点上均表现出优异性能。尤其值得一提的是,Ctrl-X支持利用任意模态的任意条件图像实现新颖的结构与外观控制,与现有工作相比图像质量和外观迁移效果更优,还能为任意T2I和文本到视频(T2V)扩散模型提供即插即用的即时功能。结果概览可参见我们的项目页面:https://genforce.github.io/ctrl-x

英文摘要:

Recent controllable generation approaches such as FreeControl and Diffusion Self-Guidance bring fine-grained spatial and appearance control to text-to-image (T2I) diffusion models without training auxiliary modules. However, these methods optimize the latent embedding for each type of score function with longer diffusion steps, making the generation process time-consuming and limiting their flexibility and use. This work presents Ctrl-X, a simple framework for T2I diffusion controlling structure and appearance without additional training or guidance. Ctrl-X designs feed-forward structure control to enable the structure alignment with a structure image and semantic-aware appearance transfer to facilitate the appearance transfer from a user-input image. Extensive qualitative and quantitative experiments illustrate the superior performance of Ctrl-X on various condition inputs and model checkpoints. In particular, Ctrl-X supports novel structure and appearance control with arbitrary condition images of any modality, exhibits superior image quality and appearance transfer compared to existing works, and provides instant plug-and-play functionality to any T2I and text-to-video (T2V) diffusion model. See our project page for an overview of the results: https://genforce.github.io/ctrl-x

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑