SpaText：用于可控图像生成的时空文本表示
SpaText: Spatio-Textual Representation for Controllable Image Generation
AI总结：
SpaText提出一种基于CLIP的时空文本表示，实现开放词汇的细粒度场景控制图像生成，在两种扩散模型上达到最先进性能。
摘要：
近期的文本到图像扩散模型能够生成具有空前质量的令人信服的结果。然而，几乎不可能以细粒度方式控制不同区域/对象的形状或其布局。先前尝试提供此类控制的方法因依赖固定标签集而受到阻碍。为此，我们提出了SpaText——一种使用开放词汇场景控制的文本到图像生成新方法。除了描述整个场景的全局文本提示外，用户还提供一个分割图，其中每个感兴趣区域由自由形式的自然语言描述进行标注。由于缺乏对图像中每个区域具有详细文本描述的大规模数据集，我们选择利用当前的大规模文本到图像数据集，并基于一种新颖的基于CLIP的时空文本表示来构建我们的方法，并展示了其在两种最先进的扩散模型（基于像素和基于潜变量）上的有效性。此外，我们展示了如何将扩散模型中的无分类器引导方法扩展到多条件情况，并提出了一种替代的加速推理算法。最后，我们提供了几种自动评估指标，并将它们与FID分数和用户研究一起使用，以评估我们的方法，并表明它在具有自由形式文本场景控制的图像生成方面达到了最先进的结果。
英文摘要：
Recent text-to-image diffusion models are able to generate convincing results of unprecedented quality. However, it is nearly impossible to control the shapes of different regions/objects or their layout in a fine-grained fashion. Previous attempts to provide such controls were hindered by their reliance on a fixed set of labels. To this end, we present SpaText - a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map where each region of interest is annotated by a free-form natural language description. Due to lack of large-scale datasets that have a detailed textual description for each region in the image, we choose to leverage the current large-scale text-to-image datasets and base our approach on a novel CLIP-based spatio-textual representation, and show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-based. In addition, we show how to extend the classifier-free guidance method in diffusion models to the multi-conditional case and present an alternative accelerated inference algorithm. Finally, we offer several automatic evaluation metrics and use them, in addition to FID scores and a user study, to evaluate our method and show that it achieves state-of-the-art results on image generation with free-form textual scene control.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
