迈向图像字幕的检索增强架构
Towards Retrieval-Augmented Architectures for Image Captioning
AI总结：
本研究提出利用外部kNN记忆的检索增强图像字幕模型，通过视觉相似性检索和kNN增强语言模型提升生成质量，并在COCO和nocaps数据集上验证了其有效性。
摘要：
图像字幕模型的目标是通过生成准确反映输入图像内容的自然语言描述，弥合视觉与语言模态之间的差距。近年来，研究人员利用基于深度学习的方法，在视觉特征提取和多模态连接设计方面取得了进展，以应对这一任务。本研究提出了一种新颖的方法，用于开发利用外部kNN记忆来改进生成过程的图像字幕模型。具体而言，我们提出了两种模型变体，它们整合了一个基于视觉相似性的知识检索组件、一个用于表示输入图像的可微编码器，以及一个基于上下文线索和从外部记忆检索的文本预测标记的kNN增强语言模型。我们在COCO和nocaps数据集上实验验证了我们的方法，并证明整合显式外部记忆可以显著提高字幕质量，尤其是在检索语料库更大时。这项工作为检索增强字幕模型提供了有价值的见解，并为更大规模地改进图像字幕开辟了新途径。
英文摘要：
The objective of image captioning models is to bridge the gap between the visual and linguistic modalities by generating natural language descriptions that accurately reflect the content of input images. In recent years, researchers have leveraged deep learning-based models and made advances in the extraction of visual features and the design of multimodal connections to tackle this task. This work presents a novel approach towards developing image captioning models that utilize an external kNN memory to improve the generation process. Specifically, we propose two model variants that incorporate a knowledge retriever component that is based on visual similarities, a differentiable encoder to represent input images, and a kNN-augmented language model to predict tokens based on contextual cues and text retrieved from the external memory. We experimentally validate our approach on COCO and nocaps datasets and demonstrate that incorporating an explicit external memory can significantly enhance the quality of captions, especially with a larger retrieval corpus. This work provides valuable insights into retrieval-augmented captioning models and opens up new avenues for improving image captioning at a larger scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
