CgT-GAN: 用于图像字幕生成的CLIP引导文本GAN
CgT-GAN: CLIP-guided Text GAN for Image Captioning
AI总结：
针对无标注图像字幕任务中纯文本训练范式的缺陷，提出融合视觉模态的CgT-GAN，结合对抗训练与CLIP语义奖励，在三个子任务上全面超越SOTA。
摘要：
大规模视觉语言预训练模型对比语言-图像预训练（Contrastive Language-Image Pre-training, CLIP）显著提升了无人工标注图像-字幕对场景下的图像字幕生成效果。近期先进的无人工标注CLIP基图像字幕方法遵循纯文本训练范式，即从共享嵌入空间重构文本。然而，这类方法受限于训练/推理差距，或是需要存储海量文本嵌入。鉴于现实世界中获取图像十分容易，我们提出CLIP引导文本GAN（CgT-GAN），将图像纳入训练过程，使模型能够“看见”真实的视觉模态。具体而言，我们采用对抗训练让CgT-GAN模仿外部文本语料的短语表达，并使用基于CLIP的奖励提供语义引导。字幕生成器的联合奖励由两部分构成：GAN判别器计算的字幕符合人类语言的自然度奖励，以及基于CLIP的奖励模块计算的语义引导奖励。除了将余弦相似度作为语义引导奖励（即CLIP-cos），我们还提出了一种名为CLIP-agg的新型语义引导奖励，它通过注意力机制聚合整个语料库，将生成的字幕与加权文本嵌入对齐。在三个子任务（ZS-IC、In-UIC和Cross-UIC）上的实验结果表明，CgT-GAN在所有指标上均显著优于当前最优方法。代码可在https://github.com/Lihr747/CgtGAN获取。
英文摘要：
The large-scale visual-language pre-trained model, Contrastive Language-Image Pre-training (CLIP), has significantly improved image captioning for scenarios without human-annotated image-caption pairs. Recent advanced CLIP-based image captioning without human annotations follows a text-only training paradigm, i.e., reconstructing text from shared embedding space. Nevertheless, these approaches are limited by the training/inference gap or huge storage requirements for text embeddings. Given that it is trivial to obtain images in the real world, we propose CLIP-guided text GAN (CgT-GAN), which incorporates images into the training process to enable the model to "see" real visual modality. Particularly, we use adversarial training to teach CgT-GAN to mimic the phrases of an external text corpus and CLIP-based reward to provide semantic guidance. The caption generator is jointly rewarded based on the caption naturalness to human language calculated from the GAN's discriminator and the semantic guidance reward computed by the CLIP-based reward module. In addition to the cosine similarity as the semantic guidance reward (i.e., CLIP-cos), we further introduce a novel semantic guidance reward called CLIP-agg, which aligns the generated caption with a weighted text embedding by attentively aggregating the entire corpus. Experimental results on three subtasks (ZS-IC, In-UIC and Cross-UIC) show that CgT-GAN outperforms state-of-the-art methods significantly across all metrics. Code is available at https://github.com/Lihr747/CgtGAN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
