仅文本训练的图像描述生成：结合检索增强与模态差距校正
Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction
AI总结：
TOMCap通过检索增强和模态差距校正，实现无需对齐图像-文本对的纯文本训练图像描述生成。
摘要：
图像描述生成已引起自然语言处理和计算机视觉领域的广泛关注。旨在减少对精心编纂数据的依赖，一些研究探索了在没有人工标注的图像-文本对的情况下进行图像描述生成，尽管现有方法仍无法超越完全监督的方法。本文提出了TOMCap，即一种改进的纯文本训练方法，能够在不需对齐的图像-描述对的情况下进行描述生成。该方法基于提示预训练语言模型解码器，使用经过处理以减少模态差距的CLIP表示信息。我们特别测试了检索示例和潜在向量表示的结合使用，以引导生成过程。通过广泛的实验，我们证明TOMCap优于其他无训练和纯文本方法。我们还分析了不同检索增强和模态差距减少组件配置的影响。
英文摘要：
Image captioning has drawn considerable attention from the natural language processing and computer vision fields. Aiming to reduce the reliance on curated data, several studies have explored image captioning without any humanly-annotated image-text pairs for training, although existing methods are still outperformed by fully supervised approaches. This paper proposes TOMCap, i.e., an improved text-only training method that performs captioning without the need for aligned image-caption pairs. The method is based on prompting a pre-trained language model decoder with information derived from a CLIP representation, after undergoing a process to reduce the modality gap. We specifically tested the combined use of retrieved examples of captions, and latent vector representations, to guide the generation process. Through extensive experiments, we show that TOMCap outperforms other training-free and text-only methods. We also analyze the impact of different choices regarding the configuration of the retrieval-augmentation and modality gap reduction components.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
