RORPCap：用于图像描述的基于检索的物体与关系提示
RORPCap: Retrieval-based Objects and Relations Prompt for Image Captioning
AI总结：
提出RORPCap模型，利用物体与关系提取模型和基于Mamba的映射网络生成提示嵌入与视觉-文本嵌入，输入GPT-2生成描述，在MS-COCO上以最短训练时间达到与检测器及GCN模型相当的性能。
摘要：
图像描述旨在以开放形式为输入图像生成自然语言描述。为了准确生成与图像相关的描述，图像描述中的一个关键步骤是识别物体并理解它们在图像中的关系。现代方法通常利用物体检测器或将检测器与图卷积网络（GCN）结合。然而，这些模型存在检测信息冗余、GCN构建困难以及训练成本高的问题。为了解决这些问题，受图像-文本检索可以为输入图像提供丰富语义信息这一事实的启发，提出了一种用于图像描述的基于检索的物体与关系提示（RORPCap）。RORPCap采用物体与关系提取模型从图像中提取物体和关系词。然后将这些词并入预定义的提示模板中，并编码为提示嵌入。接着，设计了一个基于Mamba的映射网络，将CLIP提取的图像嵌入快速映射为视觉-文本嵌入。最后，将生成的提示嵌入和视觉-文本嵌入连接起来，形成文本增强的特征嵌入，并将其输入GPT-2模型以生成描述。在广泛使用的MS-COCO数据集上进行的大量实验表明，RORPCap在交叉熵损失训练下仅需2.6小时，在“Karpathy”测试集上实现了120.5%的CIDEr分数和22.0%的SPICE分数。RORPCap以最短的训练时间实现了与基于检测器和基于GCN的模型相当的性能指标，并展示了其作为图像描述替代方案的潜力。
英文摘要：
Image captioning aims to generate natural language descriptions for input images in an open-form manner. To accurately generate descriptions related to the image, a critical step in image captioning is to identify objects and understand their relations within the image. Modern approaches typically capitalize on object detectors or combine detectors with Graph Convolutional Network (GCN). However, these models suffer from redundant detection information, difficulty in GCN construction, and high training costs. To address these issues, a Retrieval-based Objects and Relations Prompt for Image Captioning (RORPCap) is proposed, inspired by the fact that image-text retrieval can provide rich semantic information for input images. RORPCap employs an Objects and relations Extraction Model to extract object and relation words from the image. These words are then incorporate into predefined prompt templates and encoded as prompt embeddings. Next, a Mamba-based mapping network is designed to quickly map image embeddings extracted by CLIP to visual-text embeddings. Finally, the resulting prompt embeddings and visual-text embeddings are concatenated to form textual-enriched feature embeddings, which are fed into a GPT-2 model for caption generation. Extensive experiments conducted on the widely used MS-COCO dataset show that the RORPCap requires only 2.6 hours under cross-entropy loss training, achieving 120.5% CIDEr score and 22.0% SPICE score on the "Karpathy" test split. RORPCap achieves comparable performance metrics to detector-based and GCN-based models with the shortest training time and demonstrates its potential as an alternative for image captioning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
