arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.14636cs.CV
收藏

通过交叉注意力控制实现免费的局部文本到图像生成

Localized Text-to-Image Generation for Free via Cross Attention Control

Carnegie Mellon University(卡内基梅隆大学);Bosch Center for AI(博世人工智能中心)

Yutong He, Ruslan Salakhutdinov, J. Zico Kolter

AI总结:

本文提出交叉注意力控制(CAC),在无需额外训练或推理时间的情况下,通过控制推理时的交叉注意力图实现局部文本到图像生成,提升了多种位置信息下的生成性能和模型组合能力。

摘要:

尽管文本到图像生成模型取得了巨大成功,但局部文本到图像生成(即在保持整体生成一致性的同时,在图像特定位置生成对象或特征)仍然需要显式训练或大量额外的推理时间。在这项工作中,我们展示了仅通过在推理过程中控制交叉注意力图即可实现局部生成。无需额外训练、模型架构修改或增加推理时间,我们提出的交叉注意力控制(CAC)为标准文本到图像模型提供了新的开放词汇定位能力。在推理时部署时,CAC还能增强已针对局部生成训练过的模型。此外,为了自动评估局部文本到图像生成性能,我们开发了一套使用大型预训练识别模型的标准化评估套件。我们的实验表明,CAC在使用从边界框到语义分割图等多种类型的位置信息时均能提升局部生成性能,并增强了最先进的文本到图像生成模型的组合能力。

英文摘要:

Despite the tremendous success in text-to-image generative models, localized text-to-image generation (that is, generating objects or features at specific locations in an image while maintaining a consistent overall generation) still requires either explicit training or substantial additional inference time. In this work, we show that localized generation can be achieved by simply controlling cross attention maps during inference. With no additional training, model architecture modification or inference time, our proposed cross attention control (CAC) provides new open-vocabulary localization abilities to standard text-to-image models. CAC also enhances models that are already trained for localized generation when deployed at inference time. Furthermore, to assess localized text-to-image generation performance automatically, we develop a standardized suite of evaluations using large pretrained recognition models. Our experiments show that CAC improves localized generation performance with various types of location information ranging from bounding boxes to semantic segmentation maps, and enhances the compositional capability of state-of-the-art text-to-image generative models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑