超越视觉：基于多模态检索的上下文丰富图像描述
Beyond Vision: Contextually Enriched Image Captioning with Multi-Modal Retrieval
AI总结：
本文提出一种多模态检索方法，通过整合外部文本知识生成更丰富的上下文图像描述，提升视觉-文本理解能力。
摘要：
现实中的图像描述往往缺乏上下文深度，忽略了诸如事件背景、时间线索、结果和命名实体等无法通过视觉辨识的关键细节。这种差距限制了图像理解在新闻、教育和数字档案等领域的有效性，这些领域需要更丰富、更有信息量的描述。为此，我们提出了一种多模态流程，通过外部文本知识增强视觉输入。我们的系统使用BEIT-3（Flickr30k-384和COCO-384）和SigLIP So-384检索语义相似的图像，使用ORB和SIFT进行几何对齐进行重新排序，并通过语义搜索从相关文章中提取上下文信息。经过微调的Qwen3模型与QLoRA结合，将此上下文与由Instruct BLIP（Vicuna-7B）生成的基础描述相结合，生成事件丰富、上下文感知的描述。在OpenEvents v1数据集上评估，我们的方法生成的描述比传统方法更具信息量，显示出在需要更深入视觉-文本理解的现实应用中的强大潜力。
英文摘要：
Real-world image captions often lack contextual depth, omitting crucial details such as event background, temporal cues, outcomes, and named entities that are not visually discernible. This gap limits the effectiveness of image understanding in domains like journalism, education, and digital archives, where richer, more informative descriptions are essential. To address this, we propose a multimodal pipeline that augments visual input with external textual knowledge. Our system retrieves semantically similar images using BEIT-3 (Flickr30k-384 and COCO-384) and SigLIP So-384, reranks them using ORB and SIFT for geometric alignment, and extracts contextual information from related articles via semantic search. A fine-tuned Qwen3 model with QLoRA then integrates this context with base captions generated by Instruct BLIP (Vicuna-7B) to produce event-enriched, context-aware descriptions. Evaluated on the OpenEvents v1 dataset, our approach generates significantly more informative captions compared to traditional methods, showing strong potential for real-world applications requiring deeper visual-textual understanding
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
