arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.13153cs.CVcs.GRcs.LG
收藏

Directed Diffusion:通过注意力引导直接控制物体放置

Directed Diffusion: Direct Control of Object Placement through Attention Guidance

NVIDIA Research(英伟达研究院);Victoria University of Wellington(惠灵顿维多利亚大学);Google Research(谷歌研究院);Google(谷歌)

Wan-Duo Kurt Ma, J. P. Lewis, Avisek Lahiri, Thomas Leung, W. Bastiaan Kleijn

AI总结:

针对文本引导扩散模型难以按指定位置关系组合多物体的问题,本文提出 Directed Diffusion,通过优化交叉注意力图在目标位置产生激活,实现简便的高级物体放置控制。

摘要:

诸如 DALLE-2、Imagen、eDiff-I 和 Stable Diffusion 等文本引导扩散模型,仅根据描述所需图像内容的简短文本提示,就能够生成几乎无穷无尽的多样图像。在许多情况下,这些图像具有非常高的质量。然而,这些模型往往难以组合包含若干关键物体(例如角色)且这些物体处于指定位置关系的场景。正如电影和动画理论文献所认识到的那样,缺少在图像内部以及跨图像“指挥”角色和物体放置的能力,对于叙事而言至关重要。在本工作中,我们采用一种特别直接的方法来提供所需的指挥。基于提示词的交叉注意力图反映这些词所指物体空间布局这一观察,我们引入了一个优化目标,在这些交叉注意力图中的期望位置产生“激活”。所得方法朝着将文本引导扩散模型的适用范围从单张图像推广到相关图像集合(如故事书中的图像)迈出了一步。Directed Diffusion 提供了对多个物体简便的高级位置控制,同时利用现有的预训练模型,并保持被定位物体与背景之间连贯的融合。此外,它只需几行代码即可实现。

英文摘要:

Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑