EVCap：面向开放世界理解的、使用外部视觉-名称记忆的检索增强图像描述
EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension
AI总结：
本文提出 EVCap，通过可低成本更新的外部视觉-名称记忆检索物体名称并提示 LLM，以轻量模型实现开放世界图像描述。
摘要：
基于大型语言模型（LLM）的图像描述具备描述训练数据中未明确观察到的物体的能力；然而，新物体频繁出现，因此必须持续维护最新的物体知识以实现开放世界理解。我们不再依赖大量数据和/或扩大网络参数，而是提出一种高效的检索增强图像描述方法 EVCap，利用从外部视觉-名称记忆（External Visual–name memory）中检索到的物体名称来提示 LLM。我们利用物体的视觉外观和名称构建不断变化的物体知识记忆，从而能够：（i）以极低代价更新记忆；（ii）通过使用一个轻量级且训练快速的模型，轻松地用检索到的物体名称增强 LLM。我们的模型仅在 COCO 数据集上训练，无需额外微调或重新训练即可适应域外场景。在基准数据集和合成的违反常识数据上进行的实验表明，EVCap 仅有 3.97M 可训练参数，却表现出优于其他基于冻结预训练 LLM 的方法的性能；其性能也与需要大量训练的专用 SOTA 模型具有竞争力。
英文摘要：
Large language models (LLMs)-based image captioning has the capability of describing objects not explicitly observed in training data; yet novel objects occur frequently, necessitating the requirement of sustaining up-to-date object knowledge for open-world comprehension. Instead of relying on large amounts of data and/or scaling up network parameters, we introduce a highly effective retrieval-augmented image captioning method that prompts LLMs with object names retrieved from External Visual--name memory (EVCap). We build ever-changing object knowledge memory using objects' visuals and names, enabling us to (i) update the memory at a minimal cost and (ii) effortlessly augment LLMs with retrieved object names by utilizing a lightweight and fast-to-train model. Our model, which was trained only on the COCO dataset, can adapt to out-of-domain without requiring additional fine-tuning or re-training. Our experiments conducted on benchmarks and synthetic commonsense-violating data show that EVCap, with only 3.97M trainable parameters, exhibits superior performance compared to other methods based on frozen pre-trained LLMs. Its performance is also competitive to specialist SOTAs that require extensive training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
