arXiv 2604.019152026-04-02cs.CV

通过知识引导的空间提示增强医学视觉 grounding

Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts

School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Yong Yoo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）

Yifan Gao, Tao Zhou, Yi Zhou, Ke Zou, Yizhe Zhang, Huazhu Fu

备注：: 10 pages, 6 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出KnowMVG框架，通过知识增强提示和全局局部注意力机制提升医学视觉 grounding的精度，实验显示在四个基准上优于现有方法。

摘要：

医学视觉 grounding（MVG）旨在从自由文本放射科报告中识别诊断相关短语，并在医学图像中定位对应区域，为临床决策提供可解释的视觉证据。尽管近期视觉语言模型（VLMs）表现出有希望的多模态推理能力，但其 grounding 仍缺乏足够的空间精度，主要由于仅依赖潜在嵌入时缺乏显式的定位先验。本文从注意力角度分析这一限制，提出KnowMVG，一种用于VLMs中MVG的知识先验和全局局部注意力增强框架，通过解码过程中显式加强空间意识。具体而言，我们提出一种知识增强的提示策略，将与短语相关的医学知识编码为紧凑的嵌入，同时结合全局局部注意力，共同利用粗粒度全局信息和精细局部线索指导精确区域定位。该设计在不引入额外文本推理开销的情况下，连接了高层语义理解和细粒度视觉感知。在四个MVG基准上的广泛实验表明，我们的KnowMVG在AP50和mIoU上分别比现有方法提升了3.0%和2.6%。定性及消融研究进一步验证了每个组件的有效性。

英文摘要：

Medical Visual Grounding (MVG) aims to identify diagnostically relevant phrases from free-text radiology reports and localize their corresponding regions in medical images, providing interpretable visual evidence to support clinical decision-making. Although recent Vision-Language Models (VLMs) exhibit promising multimodal reasoning ability, their grounding remains insufficient spatial precision, largely due to a lack of explicit localization priors when relying solely on latent embeddings. In this work, we analyze this limitation from an attention perspective and propose KnowMVG, a Knowledge-prior and global-local attention enhancement framework for MVG in VLMs that explicitly strengthens spatial awareness during decoding. Specifically, we present a knowledge-enhanced prompting strategy that encodes phrase related medical knowledge into compact embeddings, together with a global-local attention that jointly leverages coarse global information and refined local cues to guide precise region localization. localization. This design bridges high-level semantic understanding and fine-grained visual perception without introducing extra textual reasoning overhead. Extensive experiments on four MVG benchmarks demonstrate that our KnowMVG consistently outperforms existing approaches, achieving gains of 3.0% in AP50 and 2.6% in mIoU over prior state-of-the-art methods. Qualitative and ablation studies further validate the effectiveness of each component.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。