探索基于检索增强伪句子生成的无标注图像描述生成
Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generation
AI总结：
本文提出RaPSG策略，通过蒸馏大型预训练模型知识并从不匹配语料库检索相关区域描述生成高质量伪句子，结合流畅性过滤器和CLIP指导，在零样本、无监督等多种设置下优于SOTA模型。
摘要：
近期，在没有标注的图像-句子对的情况下训练图像描述器引起了广泛关注。先前的方法由于使用不匹配的语料库导致伪标注不准确，或依赖资源密集型的预训练，从而面临局限性。为了缓解这些挑战，我们提出了一种新策略：将来自大型预训练模型（LPMs）的先验知识进行蒸馏并作为监督，同时集成检索过程以进一步增强其有效性。具体而言，我们引入了检索增强伪句子生成（RaPSG），它能从不匹配的语料库中高效检索高度相关的短区域描述，并通过LPMs利用它们生成多种高质量的伪句子。此外，我们引入了流畅性过滤器和CLIP指导目标来增强对比信息学习。实验结果表明，我们的方法在零样本、无监督、半监督和跨域等多种设置下均优于SOTA描述模型。代码可在 https://github.com/Zhiyuan-Li-John/RaPSG 获取。
英文摘要：
Recently, training an image captioner without annotated image-sentence pairs has gained traction. Previous methods have faced limitations due to either using mismatched corpora for inaccurate pseudo annotations or relying on resource-intensive pre-training. To alleviate these challenges, we propose a new strategy where the prior knowledge from large pre-trained models (LPMs) is distilled and leveraged as supervision, and a retrieval process is integrated to further reinforce its effectiveness. Specifically, we introduce Retrieval-augmented Pseudo Sentence Generation (RaPSG), which can efficiently retrieve highly relevant short region descriptions from the mismatching corpora and use them to generate a variety of high-quality pseudo sentences via LPMs. Additionally, we introduce a fluency filter and a CLIP guidance objective to enhance contrastive information learning. Experimental results indicate that our method outperforms SOTA captioning models across various settings including zero-shot, unsupervised, semi-supervised, and cross-domain scenarios. Code is available at: https://github.com/Zhiyuan-Li-John/RaPSG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
