arXiv 2404.067982025-08-06cs.CV

不确定性感知的医学诊断短语识别与定位

Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding

College of Computer Science, Sichuan University（四川大学计算机学院）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Department of Radiology, West China Hospital, Sichuan University（四川大学华西医院放射科）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Department of Mathematics, Sichuan University（四川大学数学系）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区网络科学与技术学院）；Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore and the Singapore Eye Research Institute, Singapore National Eye Centre（新加坡国立大学 Yong Loo Lin 医学院眼科系及新加坡眼科研所、新加坡国家眼科中心）

Ke Zou, Yang Bai, Bo Liu, Yidi Chen, Zhihao Chen, Yang Zhou, Xuedong Yuan, Meng Wang, Xiaojing Shen, Xiaochun Cao, Yih Chung Tham, Huazhu Fu

备注：: 17 pages, 6 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出医学报告定位新任务及uMedGround框架，利用多模态大模型和不确定性感知预测，直接从报告识别诊断短语并生成定位框，优于现有方法，并适用于视觉问答和类别定位。

摘要：

医学短语定位对于根据短语查询识别医学图像中的相关区域至关重要，有助于实现准确的图像分析和诊断。然而，当前方法依赖人工从医学报告中提取关键短语，降低了效率并增加了临床医生的工作量。此外，缺乏模型置信度估计限制了临床信任和可用性。在本文中，我们引入了一项名为医学报告定位（MRG）的新任务，旨在以端到端的方式直接从医学报告中识别诊断短语及其对应的定位框。为了应对这一挑战，我们提出了uMedGround，一个稳健且可靠的框架，利用多模态大语言模型通过将唯一标记<BOX>嵌入词汇表来预测诊断短语，以增强检测能力。视觉编码器-解码器处理嵌入的标记和输入图像以生成定位框。关键的是，uMedGround整合了不确定性感知的预测模型，显著提高了定位预测的稳健性和可靠性。实验结果表明，uMedGround优于最先进的医学短语定位方法和微调的大型视觉-语言模型，验证了其有效性和可靠性。本研究是对MRG任务的开拓性探索，标志着该领域的首次尝试。此外，我们展示了uMedGround在医学视觉问答和基于类别的定位任务中的适用性，在这些任务中，它突出显示与关键诊断短语对齐的视觉证据，支持临床医生解释各种类型的文本输入，包括自由文本报告、视觉问答查询和类别标签。

英文摘要：

Medical phrase grounding is crucial for identifying relevant regions in medical images based on phrase queries, facilitating accurate image analysis and diagnosis. However, current methods rely on manual extraction of key phrases from medical reports, reducing efficiency and increasing the workload for clinicians. Additionally, the lack of model confidence estimation limits clinical trust and usability. In this paper, we introduce a novel task called Medical Report Grounding (MRG), which aims to directly identify diagnostic phrases and their corresponding grounding boxes from medical reports in an end-to-end manner. To address this challenge, we propose uMedGround, a robust and reliable framework that leverages a multimodal large language model to predict diagnostic phrases by embedding a unique token, <BOX>, into the vocabulary to enhance detection capabilities. A vision encoder-decoder processes the embedded token and input image to generate grounding boxes. Critically, uMedGround incorporates an uncertainty-aware prediction model, significantly improving the robustness and reliability of grounding predictions. Experimental results demonstrate that uMedGround outperforms state-of-the-art medical phrase grounding methods and fine-tuned large visual-language models, validating its effectiveness and reliability. This study represents a pioneering exploration of the MRG task, marking the first-ever endeavor in this domain. Additionally, we demonstrate the applicability of uMedGround in medical visual question answering and class-based localization tasks, where it highlights visual evidence aligned with key diagnostic phrases, supporting clinicians in interpreting various types of textual inputs, including free-text reports, visual question answering queries, and class labels.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。