arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09162cs.CV
收藏

EMMA:你的文本到图像扩散模型可以秘密接受多模态提示

EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Prompts

Nanyang Technological University(南洋理工大学);Tencent(腾讯)

Yucheng Han, Rui Wang, Chi Zhang, Juntao Hu, Pei Cheng, Bin Fu, Hanwang Zhang

AI总结:

本文提出基于 ELLA 的 EMMA,通过多模态特征连接器并冻结原 T2I 模型,使其无需混合提示训练即可接受并平衡文本与参考外观等多模态条件。

摘要:

近期图像生成的进展使得根据文本条件创建高质量图像成为可能。然而,当面对多模态条件(例如文本结合参考外观)时,现有方法难以有效平衡多种条件,通常表现出对某一种模态的偏好。为应对这一挑战,我们提出 EMMA,这是一种接受多模态提示的新型图像生成模型,构建于最先进的文本到图像(T2I)扩散模型 ELLA 之上。EMMA 通过创新的多模态特征连接器设计,在文本之外无缝纳入额外模态以引导图像生成,该设计利用特殊注意力机制有效整合文本信息与补充模态信息。通过冻结原始 T2I 扩散模型中的所有参数,仅调整部分额外层,我们揭示了一个有趣发现:预训练 T2I 扩散模型能够秘密接受多模态提示。这一特性便于轻松适配不同的现有框架,使 EMMA 成为灵活且有效的工具,用于生成个性化、具备情境感知的图像乃至视频。此外,我们提出一种组装已学习 EMMA 模块的策略,以同时根据多种模态生成图像,无需使用混合多模态提示进行额外训练。大量实验证明,EMMA 在保持生成图像的高保真度和细节方面十分有效,展现出其作为先进多模态条件图像生成任务稳健解决方案的潜力。

英文摘要:

Recent advancements in image generation have enabled the creation of high-quality images from text conditions. However, when facing multi-modal conditions, such as text combined with reference appearances, existing methods struggle to balance multiple conditions effectively, typically showing a preference for one modality over others. To address this challenge, we introduce EMMA, a novel image generation model accepting multi-modal prompts built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. EMMA seamlessly incorporates additional modalities alongside text to guide image generation through an innovative Multi-modal Feature Connector design, which effectively integrates textual and supplementary modal information using a special attention mechanism. By freezing all parameters in the original T2I diffusion model and only adjusting some additional layers, we reveal an interesting finding that the pre-trained T2I diffusion model can secretly accept multi-modal prompts. This interesting property facilitates easy adaptation to different existing frameworks, making EMMA a flexible and effective tool for producing personalized and context-aware images and even videos. Additionally, we introduce a strategy to assemble learned EMMA modules to produce images conditioned on multiple modalities simultaneously, eliminating the need for additional training with mixed multi-modal prompts. Extensive experiments demonstrate the effectiveness of EMMA in maintaining high fidelity and detail in generated images, showcasing its potential as a robust solution for advanced multi-modal conditional image generation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑