通过拼贴表示实现可控图像生成
Controllable Image Generation via Collage Representations
AI总结：
本文提出 M&Ms 拼贴条件图像生成框架，利用元素外观与位置实现无标签细粒度场景控制，并在 OI 和 MS-COCO 上验证其质量、多样性及零样本泛化优势。
摘要：
条件生成图像模型的最新进展已经带来了令人瞩目的成果。一方面，基于文本的条件模型通过利用大规模图文对数据集，取得了显著的生成质量。然而，为了实现细粒度可控性，基于文本的模型需要冗长的提示，而其中的细节可能会被模型忽略。另一方面，基于布局的条件模型也见证了重大进展。这些模型依赖边界框或分割图来进行精确的空间条件控制，并结合粗略的语义标签。然而，语义标签无法用于表达详细的外观特征。在本文中，我们通过图像拼贴来处理细粒度场景可控性问题；图像拼贴允许对所需场景以及其中对象的外观和位置进行丰富的视觉描述，而无需类别或属性标签。我们提出了“mixing and matching scenes”（M&Ms）方法，该方法包含一个对抗训练的生成图像模型，以拼贴中不同元素的外观特征和空间位置为条件，并将这些元素整合为一幅连贯的图像。我们在 OpenImages（OI）数据集上训练模型，并在源自 OI 和 MS-COCO 数据集的拼贴上进行评估。我们在 OI 数据集上的实验表明，M&Ms 在细粒度场景可控性方面优于基线，同时在图像质量和样本多样性方面具有很强的竞争力。在 MS-COCO 数据集上，我们通过在零样本 FID 指标上优于 DALL-E 来突出模型的泛化能力，尽管其使用的参数和数据少了两个数量级。基于拼贴的生成模型有潜力以高效且有效的方式推进内容创作，因为它们直观易用，并能产生高质量的生成结果。
英文摘要：
Recent advances in conditional generative image models have enabled impressive results. On the one hand, text-based conditional models have achieved remarkable generation quality, by leveraging large-scale datasets of image-text pairs. To enable fine-grained controllability, however, text-based models require long prompts, whose details may be ignored by the model. On the other hand, layout-based conditional models have also witnessed significant advances. These models rely on bounding boxes or segmentation maps for precise spatial conditioning in combination with coarse semantic labels. The semantic labels, however, cannot be used to express detailed appearance characteristics. In this paper, we approach fine-grained scene controllability through image collages which allow a rich visual description of the desired scene as well as the appearance and location of the objects therein, without the need of class nor attribute labels. We introduce "mixing and matching scenes" (M&Ms), an approach that consists of an adversarially trained generative image model which is conditioned on appearance features and spatial positions of the different elements in a collage, and integrates these into a coherent image. We train our model on the OpenImages (OI) dataset and evaluate it on collages derived from OI and MS-COCO datasets. Our experiments on the OI dataset show that M&Ms outperforms baselines in terms of fine-grained scene controllability while being very competitive in terms of image quality and sample diversity. On the MS-COCO dataset, we highlight the generalization ability of our model by outperforming DALL-E in terms of the zero-shot FID metric, despite using two magnitudes fewer parameters and data. Collage based generative models have the potential to advance content creation in an efficient and effective way as they are intuitive to use and yield high quality generations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
