Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation
Independent Researcher(独立研究者)
Athos Georgiou
备注:
21 pages, 6 figures, 8 tables. Includes ancillary files with full benchmark results and ablation studies. Code available at https://github.com/athrael-soju/Snappy
Late-interaction multimodal retrieval models like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they operate at page-level granularity, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on area efficiency. We evaluate on BBox-DocVQA with ground-truth bounding boxes. For within-page localization (given correct page retrieval), ColQwen3-4B with percentile-50 thresholding achieves 59.7% hit rate at IoU@0.5 (84.4% at IoU@0.25, 35.8% at IoU@0.7), with mean IoU of 0.569, compared to ~6.7% for random region selection. Our approach reduces context tokens by 28.8% compared to returning all OCR regions and by 52.3% compared to full-page image tokens. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation at https://github.com/athrael-soju/Snappy.
InfoDPP-PAC: Principled Patch Selection for Whole Slide Image Analysis
Motilal Nehru National Institute of Technology Allahabad(Motilal Nehru 阿拉哈巴德国立技术学院);Vision Exploration and Data Analytics (VEDAs) Lab(视觉探索与数据分析实验室(VEDAs))
Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies
The Hong Kong Polytechnic University(香港理工大学);Eastern Institute of Technology(东区技术研究所);Great Bay University(大湾大学);Northeast Forestry University(东北林业大学);National University of Singapore(新加坡国立大学)
Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Learning from the Right Patches: A Two-Stage Wavelet-Driven Masked Autoencoder for Histopathology Representation Learning
PLRI Medical Informatics Institute(PLRI医学信息学研究所);CAIMed Reserch Center(CAIMed研究中心);Hannover Medical School(汉诺威医学院);Computer Science Institute(计算机科学研究所);Leibniz University Hannover(莱比锡大学汉诺威分校);Sanford Burnham Prebys Medical Discovery Institute(桑福医疗发现研究所);Rady Children’s Institute for Genomic Medicine(拉迪儿童基因医学研究所);University of California San Diego(加州大学圣地亚哥分校)
MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation
Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院);Digital Medical Research Center, School of Basic Medical Sciences, Fudan University(复旦大学基础医学院数字医学研究中心);Shandong Computer Science Center (National Supercomputer Center in Jinan)(山东省计算中心(国家济南超级计算中心))
Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Wuhan AI Research(武汉人工智能研究院);Cardiff University(卡迪夫大学)
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Department of Mathematics and Computer Science, Faculty of Science, Alexandria University(亚历山大大学数学与计算机科学系);Sheikh Shakhbout Medical City(谢赫·沙赫布OUT医疗城)