arXiv 2607.000602026-07-02cs.CV

协同感知-推理治理：用可验证解剖证据为医学多模态大语言模型提供基础

Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

Huazhong University of Science and Technology（华中科技大学）；Imperial Global Singapore, Imperial College London（帝国理工学院新加坡全球中心）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Anhui University（安徽大学）

Rui Hao, Qiankun Li, Junyuan Mao, Linghao Meng, Dirui Xie, Dayu Tan, Zhigang Zeng

备注：: Accepted by MICCAI 2026 (Early Accept, Top 9%)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一种无需训练的证据注入框架，通过ROI引导的视觉激活调制和解剖坐标语义标记，协同校准视觉感知与文本推理，动态路由任务特定干预，有效减少医学MLLM的幻觉。

摘要：

多模态大语言模型在临床VQA和放射学报告生成中展现出强大潜力，但推理时的幻觉仍损害可信度：模型可能生成与影像证据冲突的流畅结论。现有缓解策略通常依赖额外训练、外部检索/知识库或多阶段事后验证，增加了成本和流程复杂性，且跨模型泛化性差。为此，我们提出一种全面的、无需训练的证据注入框架，通过双端证据注入系统性地缓解幻觉。利用MedSAM获取的ROI先验，我们通过ROI引导的激活调制重新校准视觉感知轨迹，同时通过将解剖坐标映射为离散语义标记作为可验证的外部记忆，锚定文本推理轨迹。然后引入任务感知动态路由器，根据任务语义选择模态特定干预，平衡感知基础与语言流畅性。我们在2个任务和5个数据集上使用LLaVA-1.5-7B、LLaVA-Med-1.5-7B、Qwen3-VL-8B/32B和InternVL-3.5-8B/38B进行系统评估。控制消融和可视化进一步验证了该框架，它在医学基准上持续优于基线，闭式准确率提升高达约6%，开式幻觉减少约35%。代码已在GitHub上公开。

英文摘要：

Multimodal large language models (MLLMs) show strong promise for clinical VQA and radiology report generation, yet inference-time hallucinations still undermine trustworthy use: models can produce fluent conclusions that conflict with imaging evidence. Existing mitigation strategies typically rely on additional training, external retrieval/knowledge bases, or multi-stage post-hoc verification, which increases cost and pipeline complexity and often generalizes poorly across models and tasks.To address this, we propose a holistic, training-free evidence-injection framework that systematically mitigates hallucinations through dual-side evidence injection. By leveraging ROI priors acquired using MedSAM in our implementation, we recalibrate the visual perception trajectory via ROI-guided activation modulation while anchoring the textual reasoning trajectory by mapping anatomical coordinates into discrete semantic tokens as verifiable external memory. Then we introduce a task-aware dynamic router to select modality-specific interventions based on task semantics, balancing perceptual grounding and linguistic fluency. We conduct systematic evaluations on 2 tasks and 5 datasets using \texttt{LLaVA-1.5-7B}, \texttt{LLaVA-Med-1.5-7B}, \texttt{Qwen3-VL-8B/32B}, and \texttt{InternVL-3.5-8B/38B}. Controlled ablations and visualizations further validate the framework, which consistently outperforms baselines across medical benchmarks, improving close-ended accuracy by up to $\sim\mathbf{6}\%\uparrow$ and reducing open-ended hallucinations by $\sim\mathbf{35}\%\downarrow$. The code has been made available on GitHub: \href{https://github.com/Henry991115/SPRG}{\textcolor{blue}{https://github.com/Henry991115/SPRG}}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。