arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.02660cs.CVcs.IR
收藏

通过补丁到区域相关性传播实现空间感知的文档检索

Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation

Independent Researcher(独立研究者)

Athos Georgiou

AI总结:

通过结合ColPali的补丁相似性与OCR区域,实现更精确的文档检索,提升RAG任务中的上下文相关性

摘要:

后期交互多模态检索模型,如ColPali,通过将页面嵌入为图像并在查询词与视觉补丁之间计算细粒度相似性来实现最先进的文档检索。然而,它们在页面级别进行操作,限制了在需要精确上下文的检索增强生成(RAG)中的实用性。相反,基于OCR的系统提取带有边界框坐标的结构化文本,但缺乏对相关性评估的语义支撑。我们提出了一种混合架构,统一了这些范式:利用ColPali的补丁级相似性评分作为OCR提取区域上的空间相关性过滤器。我们正式化了视觉转换器补丁网格与OCR边界框之间的坐标映射,引入交集指标用于相关性传播,并建立了关于面积效率的理论界限。我们在BBox-DocVQA上进行评估,使用真实标注的边界框。对于页面内定位(在正确页面检索的情况下),ColQwen3-4B在百分位数-50阈值下实现IoU@0.5的命中率为59.7%(IoU@0.25为84.4%,IoU@0.7为35.8%),平均IoU为0.569,相比随机区域选择的约6.7%。我们的方法将上下文标记减少了28.8%(相比返回所有OCR区域)和52.3%(相比全页面图像标记)。我们的方法在推理时运行,无需额外训练。我们发布了Snappy,一个开源实现,网址为https://github.com/athrael-soju/Snappy。

英文摘要:

Late-interaction multimodal retrieval models like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they operate at page-level granularity, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on area efficiency. We evaluate on BBox-DocVQA with ground-truth bounding boxes. For within-page localization (given correct page retrieval), ColQwen3-4B with percentile-50 thresholding achieves 59.7% hit rate at IoU@0.5 (84.4% at IoU@0.25, 35.8% at IoU@0.7), with mean IoU of 0.569, compared to ~6.7% for random region selection. Our approach reduces context tokens by 28.8% compared to returning all OCR regions and by 52.3% compared to full-page image tokens. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation at https://github.com/athrael-soju/Snappy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑