知识完善视觉：一种多模态实体感知检索增强生成框架用于新闻图像描述
Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning
AI总结：
MERGE提出了一种多模态实体感知检索增强生成框架，通过构建实体中心知识库和改进跨模态对齐，提升新闻图像描述质量和命名实体识别性能。
摘要：
新闻图像描述旨在通过结合视觉内容与相关文章的上下文线索，生成具有新闻报道信息的描述。尽管近年来取得了进展，但现有方法在三个关键挑战上仍存在问题：（1）信息覆盖不完整，（2）跨模态对齐薄弱，（3）视觉-实体定位不优。为了解决这些问题，我们引入了MERGE，这是首个多模态实体感知检索增强生成框架，用于新闻图像描述。MERGE构建了一个以实体为中心的多模态知识库（EMKB），整合了文本、视觉和结构化知识，使背景检索更加丰富。它通过多阶段假设-描述策略改进跨模态对齐，并通过由图像内容引导的动态检索来增强视觉-实体匹配。在GoodNews和NYTimes800k上的广泛实验表明，MERGE显著优于最先进的基线方法，在描述质量方面CIDEr提升+6.84和+1.16，在命名实体识别方面F1分数提升+4.14和+2.64。值得注意的是，MERGE还能够很好地推广到未见的Visual News数据集，在CIDEr上达到+20.17，在F1分数上达到+6.22，展示了强大的鲁棒性和领域适应性。
英文摘要：
News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
