发表机构
Northeastern University; Tsinghua University; Peking University（东北大学; 清华大学; 北京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对视觉文档检索中现有监督信号的局限，本文提出ConceptFormer框架，以自适应潜在概念为中间表示衔接语义鸿沟，在基准测试中较最强基线实现了16.7%、22.1%的NDCG@10相对提升，性能优异。
AI 中文摘要
视觉文档检索是多模态检索增强生成的关键组成部分，旨在从文档集合中识别与查询相关的页面，其中证据分布在文本、布局、图表和视觉结构中。近期更细粒度监督的工作主要依赖文本描述或局部视觉区域作为证据代理，但这类监督信号要么会忽略复杂的视觉结构，要么无法提供底层证据的完整且准确的表示。为解决这些局限，我们提出ConceptFormer，一种用于视觉文档检索的潜在概念表示学习框架。ConceptFormer将与查询相关的证据建模为连续的、以查询为条件的潜在概念，明确衔接局部视觉证据与语义相关性，无需文本中间表示或直接依赖原始视觉标注。训练期间，ConceptFormer采用强视觉-语言模型动态确定潜在概念token的数量，并将这些概念作为中间表示衔接查询与文档间的语义鸿沟，进而引导嵌入空间的学习。在多个视觉文档检索基准上的实验表明，ConceptFormer相较于最强视觉检索基线和最强基于OCR的文本检索基线，在平均NDCG@10上分别实现了16.7%和22.1%的相对提升。进一步分析显示，潜在概念可有效衔接局部视觉证据与语义相关性，使检索器既能捕捉细粒度文本线索，又能捕获复杂的文档级视觉结构，同时保持强大的检索对齐能力。代码和数据可在指定URL获取。
英文摘要
Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.
