arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04834cs.CV
收藏

FlexEControl:面向文本到图像生成的灵活高效多模态控制

FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation

UC Santa Cruz(加州大学圣克鲁兹分校);Amazon(亚马逊公司);UNC Chapel Hill(北卡罗来纳大学教堂山分校);Rice University(莱斯大学);University of California, Los Angeles(加州大学洛杉矶分校)

Xuehai He, Jian Zheng, Jacob Zhiyuan Fang, Robinson Piramuthu, Mohit Bansal, Vicente Ordonez, Gunnar A Sigurdsson, Nanyun Peng, Xin Eric Wang

AI总结:

FlexEControl提出权重分解策略,实现灵活高效的多模态可控文本到图像生成,相比Uni-ControlNet减少41%可训练参数和30%内存,数据效率翻倍。

摘要:

可控文本到图像(T2I)扩散模型根据文本提示和边缘图等其他模态的语义输入生成图像。然而,当前可控T2I方法通常面临效率和保真度方面的挑战,尤其是在同时基于来自相同或不同模态的多个输入进行条件生成时。本文提出了一种新颖的灵活高效方法FlexEControl,用于可控T2I生成。FlexEControl的核心是一种独特的权重分解策略,该策略能够简化各种输入类型的集成。这种方法不仅增强了生成图像对控制的保真度,还显著降低了通常与多模态条件生成相关的计算开销。与Uni-ControlNet相比,我们的方法可训练参数减少了41%,内存使用量减少了30%。此外,它还将数据效率提高了一倍,并能在多种模态的多个输入条件引导下灵活生成图像。

英文摘要:

Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current controllable T2I methods commonly face challenges related to efficiency and faithfulness, especially when conditioning on multiple inputs from either the same or diverse modalities. In this paper, we propose a novel Flexible and Efficient method, FlexEControl, for controllable T2I generation. At the core of FlexEControl is a unique weight decomposition strategy, which allows for streamlined integration of various input types. This approach not only enhances the faithfulness of the generated image to the control, but also significantly reduces the computational overhead typically associated with multimodal conditioning. Our approach achieves a reduction of 41% in trainable parameters and 30% in memory usage compared with Uni-ControlNet. Moreover, it doubles data efficiency and can flexibly generate images under the guidance of multiple input conditions of various modalities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑