文本到图像扩散模型的零样本空间布局条件化
Zero-shot spatial layout conditioning for text-to-image diffusion models
AI总结：
本文提出ZestGuide，一种零样本分割引导方法，通过利用交叉注意力层的隐式分割图，将文本到图像扩散模型的生成过程与输入掩码对齐，无需额外训练，在COCO数据集上较先前方法提升5至10个mIoU点。
摘要：
大规模文本到图像扩散模型显著改进了生成式图像建模的技术水平，并提供了一种直观且强大的用户界面来驱动图像生成过程。使用文本表达空间约束（例如将特定物体放置在特定位置）非常繁琐；而当前基于文本的图像生成模型无法准确遵循此类指令。在本文中，我们考虑根据与图像画布上的片段相关联的文本生成图像，这结合了直观的自然语言界面与对生成内容的精确空间控制。我们提出了ZestGuide，这是一种零样本分割引导方法，可插入预训练的文本到图像扩散模型中，且不需要任何额外训练。它利用可从交叉注意力层中提取的隐式分割图，并使用它们将生成过程与输入掩码对齐。我们的实验结果结合了高图像质量与生成内容与输入分割的精确对齐，并在定量和定性上均优于先前的工作，包括那些需要在带有相应分割的图像上进行训练的方法。与Paint with Words（此前在零样本分割条件化图像生成领域的最先进方法）相比，我们在COCO数据集上取得了5到10个mIoU点的提升，且FID分数相近。
英文摘要：
Large-scale text-to-image diffusion models have significantly improved the state of the art in generative image modelling and allow for an intuitive and powerful user interface to drive the image generation process. Expressing spatial constraints, e.g. to position specific objects in particular locations, is cumbersome using text; and current text-based image generation models are not able to accurately follow such instructions. In this paper we consider image generation from text associated with segments on the image canvas, which combines an intuitive natural language interface with precise spatial control over the generated content. We propose ZestGuide, a zero-shot segmentation guidance approach that can be plugged into pre-trained text-to-image diffusion models, and does not require any additional training. It leverages implicit segmentation maps that can be extracted from cross-attention layers, and uses them to align the generation with input masks. Our experimental results combine high image quality with accurate alignment of generated content with input segmentations, and improve over prior work both quantitatively and qualitatively, including methods that require training on images with corresponding segmentations. Compared to Paint with Words, the previous state-of-the art in image generation with zero-shot segmentation conditioning, we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
