arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11487cs.CV
收藏

基于文本到图像扩散模型的视觉概念驱动图像生成

Visual Concept-driven Image Generation with Text-to-Image Diffusion Model

University of British Columbia(不列颠哥伦比亚大学);Snap Inc.(Snap公司);Vector Institute for AI(向量人工智能研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)

Tanzila Rahman, Shweta Mahajan, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Leonid Sigal

AI总结:

提出概念驱动的TTI个性化框架,通过类似EM的优化过程联合学习自定义token和潜在分割掩码,利用U-Net交叉注意力和DenseCRF解耦纠缠概念,实现多概念组合的高质量图像生成。

摘要:

文本到图像(TTI)扩散模型在生成复杂且富有想象力的场景的高分辨率图像方面展现出了令人印象深刻的结果。近期的方法进一步通过个性化技术扩展了这些方法,允许使用少量样本图像图示来整合用户说明的概念(例如用户自身)。然而,生成包含多个交互概念(如人类主体)以及在一张或多张图像图示中可能相互纠缠的概念的图像的能力仍然难以实现。在这项工作中,我们提出了一个概念驱动的TTI个性化框架来解决这些核心挑战。我们建立在现有学习用户说明概念的自定义token的工作基础上,允许这些token与TTI模型中现有的文本token进行交互。然而,重要的是,为了解耦并更好地学习相关概念,我们联合学习(潜在)分割掩码,以在用户提供的图像图示中解耦这些概念。我们通过引入类似期望最大化(EM)的优化过程来实现这一点,在该过程中我们在学习自定义token和估计用户提供的图像中包含相应概念的(潜在)掩码之间交替进行。我们基于U-Net参数化的潜在扩散模型内部的交叉注意力和随后的DenseCRF优化来获取这些掩码。我们证明,这种联合交替细化能够学习到更好的概念token,并作为副产品产生更好的潜在掩码。我们通过定性和定量的方式,利用几个能够结合三个或更多纠缠概念的示例和用例,展示了所提方法的优势。

英文摘要:

Text-to-image (TTI) diffusion models have demonstrated impressive results in generating high-resolution images of complex and imaginative scenes. Recent approaches have further extended these methods with personalization techniques that allow them to integrate user-illustrated concepts (e.g., the user him/herself) using a few sample image illustrations. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. In this work, we propose a concept-driven TTI personalization framework that addresses these core challenges. We build on existing works that learn custom tokens for user-illustrated concepts, allowing those to interact with existing text tokens in the TTI model. However, importantly, to disentangle and better learn the concepts in question, we jointly learn (latent) segmentation masks that disentangle these concepts in user-provided image illustrations. We do so by introducing an Expectation Maximization (EM)-like optimization procedure where we alternate between learning the custom tokens and estimating (latent) masks encompassing corresponding concepts in user-supplied images. We obtain these masks based on cross-attention, from within the U-Net parameterized latent diffusion model and subsequent DenseCRF optimization. We illustrate that such joint alternating refinement leads to the learning of better tokens for concepts and, as a by-product, latent masks. We illustrate the benefits of the proposed approach qualitatively and quantitatively with several examples and use cases that can combine three or more entangled concepts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑