基于注意力的视觉文档检索增强
Attention Grounded Enhancement for Visual Document Retrieval
AI总结：
本文提出AGREE框架，通过多模态大语言模型的注意力机制引导检索器识别相关文档区域，提升细粒度相关性建模，实验表明在ViDoRe V2基准上显著优于传统方法。
摘要：
视觉文档检索需要理解异构和多模态内容以满足隐式信息需求。近期进展利用截图基于的文档编码结合细粒度后期交互来编码整体信息并捕捉细微对齐，显著提升检索性能。然而，检索器仍使用粗粒度全局相关性标签进行训练，未揭示哪些区域支持匹配。因此，检索器倾向于依赖表面特征，难以捕捉隐含语义连接，阻碍其处理非提取查询的能力。为提高细粒度相关性建模，我们提出注意力引导的检索增强（AGREE）框架。AGREE利用多模态大语言模型（MLLMs）的跨模态注意力作为代理监督，引导检索器识别相关文档区域。具体而言，AGREE从MLLM中提取注意力图，突出基于查询的文档区域。这些注意力分数作为局部、区域级的相关性信号。在训练过程中，AGREE将局部信号与全局文档级相关性标签结合，共同优化检索器。这种双层监督使模型不仅能学习文档是否匹配，还能学习哪些内容驱动相关性。在挑战性的视觉文档检索基准ViDoRe V2上，实验表明AGREE在平均nDCG@1和nDCG@5上分别比仅使用全局监督的基线高出12.82%和5.03%。定量和定性分析进一步表明，AGREE促进了查询术语与文档区域之间的深度对齐，超越了表面匹配，向更准确和可解释的检索迈进。我们的代码可在：https://github.com/VickiCui/AGREE获取。
英文摘要：
Visual document retrieval requires understanding heterogeneous and multi-modal content to satisfy implicit information needs. Recent advances use screenshot-based document encoding with fine-grained late interaction to encode holistic information and capture nuanced alignments, significantly improving retrieval performance. However, retrievers are still trained with coarse global relevance labels, without revealing which regions support the match. As a result, retrievers tend to rely on surface-level cues and struggle to capture implicit semantic connections, hindering their ability to handle non-extractive queries.To improve fine-grained relevance modeling, we propose a Attention-Grounded REtriever Enhancement (AGREE) framework. AGREE leverages cross-modal attention from multimodal large language models (MLLMs) as proxy supervision to guide the retriever in identifying relevant document regions. Specifically, AGREE extracts attention maps from the MLLM that highlight which document regions are attended to based on the query. These attention scores serve as local, region-level relevance signals. During training, AGREE combines local signals with the global document-level relevance label to jointly optimize the retriever. This dual-level supervision enables the model to learn not only whether documents match, but also which content drives relevance. Experiments on the challenging visual document retrieval benchmark, ViDoRe V2, show that AGREE significantly outperforms the global-supervision-only baseline by 12.82\% and 5.03\% in terms of average nDCG@1 and nDCG@5. Quantitative and qualitative analyses further demonstrate that AGREE promotes deeper alignment between query terms and document regions, moving beyond surface-level matching toward more accurate and interpretable retrieval. Our code is available at: https://github.com/VickiCui/AGREE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
