MeaCap：记忆增强的零样本图像字幕生成
MeaCap: Memory-Augmented Zero-shot Image Captioning
AI总结：
针对现有零样本图像字幕生成方法易产生幻觉或泛化性差的问题，提出记忆增强框架MeaCap，通过检索过滤关键概念和记忆增强融合分数生成高质量字幕，在多设置下达SOTA。
摘要：
无需配对图像-文本数据的零样本图像字幕生成（IC）可分为免训练和仅文本训练两类。通常，这两类方法通过整合CLIP这类预训练视觉-语言模型（用于图像-文本相似度评估）和预训练语言模型（LM，用于字幕生成）来实现零样本IC，二者的主要区别在于是否使用文本语料训练LM。现有方法虽在部分指标上取得不错表现，但普遍存在缺陷：免训练方法易产生幻觉，仅文本训练方法则常丧失泛化能力。为此，本文提出一种新型记忆增强零样本图像字幕生成框架MeaCap。具体而言，该框架配备文本存储器，引入“检索-过滤”模块获取与图像高度相关的关键概念；通过在关键词转句子的语言模型中部署我们提出的记忆增强视觉相关融合分数，MeaCap能生成以概念为中心、与图像高度一致的字幕，幻觉更少且包含更多世界知识。MeaCap框架在一系列零样本IC设置下取得了当前最优性能，代码可在https://github.com/joeyz0z/MeaCap获取。
英文摘要：
Zero-shot image captioning (IC) without well-paired image-text data can be divided into two categories, training-free and text-only-training. Generally, these two types of methods realize zero-shot IC by integrating pretrained vision-language models like CLIP for image-text similarity evaluation and a pre-trained language model (LM) for caption generation. The main difference between them is whether using a textual corpus to train the LM. Though achieving attractive performance w.r.t. some metrics, existing methods often exhibit some common drawbacks. Training-free methods tend to produce hallucinations, while text-only-training often lose generalization capability. To move forward, in this paper, we propose a novel Memory-Augmented zero-shot image Captioning framework (MeaCap). Specifically, equipped with a textual memory, we introduce a retrieve-then-filter module to get key concepts that are highly related to the image. By deploying our proposed memory-augmented visual-related fusion score in a keywords-to-sentence LM, MeaCap can generate concept-centered captions that keep high consistency with the image with fewer hallucinations and more world-knowledge. The framework of MeaCap achieves the state-of-the-art performance on a series of zero-shot IC settings. Our code is available at https://github.com/joeyz0z/MeaCap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
