RATLIP：基于循环仿射变换的生成对抗CLIP文本到图像合成
RATLIP: Generative Adversarial CLIP Text-to-Image Synthesis Based on Recurrent Affine Transformations
AI总结：
本文提出RATLIP模型，通过结合循环神经网络与条件仿射变换引入全局文本信息，并利用Shuffle Attention缓解信息遗忘，同时在生成器和判别器中采用预训练CLIP模型，显著提升了文本到图像合成的质量与一致性。
摘要：
以文本描述为条件合成高质量照片级真实图像极具挑战性。作为该任务的经典模型，生成对抗网络（GANs）常面临图像与文本描述间一致性低及合成图像丰富度不足的问题。近期，条件仿射变换（CAT），如条件批归一化和实例归一化，已被应用于GAN的不同层以控制图像内容合成。CAT是一种多层感知机，基于相邻层间的批统计量独立预测数据，其他层无法获取全局文本信息。为解决此问题，我们首先将CAT与循环神经网络（RAT）结合建模，以确保不同层能访问全局信息。随后我们在RAT间引入shuffle attention，以缓解循环神经网络信息遗忘的特性。此外，我们的生成器和判别器均利用了强大的预训练模型Clip，该模型通过在潜在空间学习多模态表征，被广泛用于建立文本与图像间的关联。判别器利用CLIP理解复杂场景的能力，准确评估生成图像的质量。我们在CUB、Oxford和CelebA-tiny数据集上进行了大量实验，证明了所提模型优于当前最先进模型。代码见https://github.com/OxygenLu/RATLIP。
英文摘要：
Synthesizing high-quality photorealistic images with textual descriptions as a condition is very challenging. Generative Adversarial Networks (GANs), the classical model for this task, frequently suffer from low consistency between image and text descriptions and insufficient richness in synthesized images. Recently, conditional affine transformations (CAT), such as conditional batch normalization and instance normalization, have been applied to different layers of GAN to control content synthesis in images. CAT is a multi-layer perceptron that independently predicts data based on batch statistics between neighboring layers, with global textual information unavailable to other layers. To address this issue, we first model CAT and a recurrent neural network (RAT) to ensure that different layers can access global information. We then introduce shuffle attention between RAT to mitigate the characteristic of information forgetting in recurrent neural networks. Moreover, both our generator and discriminator utilize the powerful pre-trained model, Clip, which has been extensively employed for establishing associations between text and images through the learning of multimodal representations in latent space. The discriminator utilizes CLIP's ability to comprehend complex scenes to accurately assess the quality of the generated images. Extensive experiments have been conducted on the CUB, Oxford, and CelebA-tiny datasets to demonstrate the superiority of the proposed model over current state-of-the-art models. The code is https://github.com/OxygenLu/RATLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
