一种用于多轮组合图像生成的基于扩散的方法
A Diffusion-based Method for Multi-turn Compositional Image Generation
AI总结：
本文提出基于扩散的CDD-ICM方法，利用CLIP编码器和门控融合机制，结合多任务学习与引导策略，在多轮组合图像生成基准上达到最优性能。
摘要：
多轮组合图像生成（M-CIG）是一项具有挑战性的任务，旨在根据修改文本对参考图像进行迭代操作。虽然现有的M-CIG方法大多基于生成对抗网络（GANs），但近期图像生成领域的进展已证明了扩散模型相对于GANs的优越性。在本文中，我们提出了一种用于M-CIG的基于扩散的方法，命名为条件去噪扩散与图像组合匹配（CDD-ICM）。我们利用CLIP作为图像和文本编码器的主干，并引入了一种最初为问答提出的门控融合机制，以在M-CIG的每一轮中组合融合参考图像和修改文本。我们引入了一种条件方案，基于融合结果生成目标图像。为了优先考虑生成目标图像的语义质量，我们在多任务学习框架中，除了条件去噪扩散（CDD）目标外，还学习了一个辅助的图像组合匹配（ICM）目标。此外，我们还执行ICM引导和无分类器引导以提高性能。实验结果表明，CDD-ICM在M-CIG的两个基准数据集（即CoDraw和i-CLEVR）上取得了最先进的结果。
英文摘要：
Multi-turn compositional image generation (M-CIG) is a challenging task that aims to iteratively manipulate a reference image given a modification text. While most of the existing methods for M-CIG are based on generative adversarial networks (GANs), recent advances in image generation have demonstrated the superiority of diffusion models over GANs. In this paper, we propose a diffusion-based method for M-CIG named conditional denoising diffusion with image compositional matching (CDD-ICM). We leverage CLIP as the backbone of image and text encoders, and incorporate a gated fusion mechanism, originally proposed for question answering, to compositionally fuse the reference image and the modification text at each turn of M-CIG. We introduce a conditioning scheme to generate the target image based on the fusion results. To prioritize the semantic quality of the generated target image, we learn an auxiliary image compositional match (ICM) objective, along with the conditional denoising diffusion (CDD) objective in a multi-task learning framework. Additionally, we also perform ICM guidance and classifier-free guidance to improve performance. Experimental results show that CDD-ICM achieves state-of-the-art results on two benchmark datasets for M-CIG, i.e., CoDraw and i-CLEVR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
