基于图条件化的图像合成：面向场景图的CLIP引导扩散模型
Image Synthesis with Graph Conditioning: CLIP-Guided Diffusion Models for Scene Graphs
AI总结：
提出一种无需中间布局预测、利用CLIP引导的扩散模型从场景图直接生成图像的新方法，通过图编码器与CLIP特征对齐及条件信号融合，在COCO-stuff和Visual Genome基准上超越现有方法。
摘要：
生成模型的进步引发了人们对在遵循特定结构约束的同时生成图像的浓厚兴趣。场景图到图像的生成就是这样一项任务，即生成与给定场景图一致的图像。然而，视觉场景的复杂性对根据场景图中指定的关系准确定位对象构成了挑战。现有方法首先预测场景布局，然后使用对抗训练从这些布局生成图像。在这项工作中，我们引入了一种从场景图生成图像的新方法，该方法无需预测中间布局。我们利用预训练的文本到图像扩散模型和CLIP引导将图知识转化为图像。为此，我们首先预训练我们的图编码器，通过基于GAN的训练将图特征与对应图像的CLIP特征对齐。此外，我们将图特征与给定场景图中存在的对象标签的CLIP嵌入融合，以创建图一致的CLIP引导条件信号。在条件输入中，对象嵌入提供图像的粗略结构，图特征提供基于对象间关系的结构对齐。最后，我们使用图一致的条件信号以及重建和CLIP对齐损失对预训练的扩散模型进行微调。大量实验表明，我们的方法在COCO-stuff和Visual Genome数据集的标准基准上优于现有方法。
英文摘要：
Advancements in generative models have sparked significant interest in generating images while adhering to specific structural guidelines. Scene graph to image generation is one such task of generating images which are consistent with the given scene graph. However, the complexity of visual scenes poses a challenge in accurately aligning objects based on specified relations within the scene graph. Existing methods approach this task by first predicting a scene layout and generating images from these layouts using adversarial training. In this work, we introduce a novel approach to generate images from scene graphs which eliminates the need of predicting intermediate layouts. We leverage pre-trained text-to-image diffusion models and CLIP guidance to translate graph knowledge into images. Towards this, we first pre-train our graph encoder to align graph features with CLIP features of corresponding images using a GAN based training. Further, we fuse the graph features with CLIP embedding of object labels present in the given scene graph to create a graph consistent CLIP guided conditioning signal. In the conditioning input, object embeddings provide coarse structure of the image and graph features provide structural alignment based on relationships among objects. Finally, we fine tune a pre-trained diffusion model with the graph consistent conditioning signal with reconstruction and CLIP alignment loss. Elaborate experiments reveal that our method outperforms existing methods on standard benchmarks of COCO-stuff and Visual Genome dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
