理解检索增强图像描述生成的检索鲁棒性
Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning
AI总结：
本文分析 SmallCap 在图像描述生成中的检索鲁棒性，发现其易复制多数检索描述中的 token，并提出多样化采样检索描述的训练策略以提升域内与跨域表现。
摘要：
用于图像描述生成的检索增强模型近期进展凸显了检索相关描述的优势：它能带来高效、轻量且具备强领域迁移能力的模型。尽管这些模型证明了检索增强的成功，但在实际应用中检索模型仍远非完美：检索到的信息有时会误导模型，导致错误生成和性能下降。本文分析了检索增强描述模型 SmallCap 的鲁棒性。分析表明，该模型对出现在大多数检索描述中的 token 很敏感，而输入归因显示这些 token 很可能被复制到生成输出中。基于这些发现，作者提出通过从更多样化的集合中采样检索描述来训练模型。这降低了模型学习复制多数 token 的概率，并同时提升了域内和跨域性能。
英文摘要：
Recent advances in retrieval-augmented models for image captioning highlight the benefit of retrieving related captions for efficient, lightweight models with strong domain-transfer capabilities. While these models demonstrate the success of retrieval augmentation, retrieval models are still far from perfect in practice: the retrieved information can sometimes mislead the model, resulting in incorrect generation and worse performance. In this paper, we analyze the robustness of a retrieval-augmented captioning model SmallCap. Our analysis shows that the model is sensitive to tokens that appear in the majority of the retrieved captions, and the input attribution shows that those tokens are likely copied into the generated output. Given these findings, we propose to train the model by sampling retrieved captions from more diverse sets. This decreases the chance that the model learns to copy majority tokens, and improves both in-domain and cross-domain performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
