arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.03458eess.IVcs.AIcs.CV
收藏

激活用于基于LLM的X光报告生成的联想式疾病感知视觉Token记忆

Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation

Anhui University(安徽大学);School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);First Affiliated Hospital of Anhui University of Chinese Medicine(安徽中医药大学第一附属医院);Peng Cheng Laboratory(鹏城实验室);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peking University(北京大学);National Engineering Laboratory for Video Technology(视频技术国家工程实验室)

Xiao Wang, Fuling Wang, Haowen Wang, Bo Jiang, Chuanfu Li, Yaowei Wang, Yonghong Tian, Jin Tang

AI总结:

针对LLM生成X光报告时疾病视觉信息利用不足的问题,本文提出联想记忆增强模型,通过激活图、疾病查询token及视觉/报告Hopfield网络关联历史信息,在三个基准上达到SOTA。

摘要:

近年来,在大语言模型的帮助下,基于X光图像的医学报告生成取得了显著进展;然而,这些模型尚未充分利用视觉图像区域中的有效信息,导致生成的报告在语言上通顺,但对关键疾病的描述不足。本文提出了一种新颖的联想记忆增强型X光报告生成模型,可有效模拟专业医生撰写医学报告的过程。该模型同时考虑全局与局部视觉信息的挖掘,并关联历史报告信息,以更好地完成当前报告的撰写。具体而言,给定一张X光图像,我们首先利用分类模型及其激活图,完成与疾病高度相关的视觉区域挖掘以及疾病查询token的学习。随后,我们采用视觉Hopfield网络为疾病相关token建立记忆关联,并采用报告Hopfield网络检索报告记忆信息。该过程基于大语言模型促进高质量报告的生成,并在多个基准数据集上取得最先进性能,包括IU X-ray、MIMIC-CXR和Chexpert Plus。本工作的源代码已发布于https://github.com/Event-AHU/Medical_Image_Analysis。

英文摘要:

X-ray image based medical report generation achieves significant progress in recent years with the help of the large language model, however, these models have not fully exploited the effective information in visual image regions, resulting in reports that are linguistically sound but insufficient in describing key diseases. In this paper, we propose a novel associative memory-enhanced X-ray report generation model that effectively mimics the process of professional doctors writing medical reports. It considers both the mining of global and local visual information and associates historical report information to better complete the writing of the current report. Specifically, given an X-ray image, we first utilize a classification model along with its activation maps to accomplish the mining of visual regions highly associated with diseases and the learning of disease query tokens. Then, we employ a visual Hopfield network to establish memory associations for disease-related tokens, and a report Hopfield network to retrieve report memory information. This process facilitates the generation of high-quality reports based on a large language model and achieves state-of-the-art performance on multiple benchmark datasets, including the IU X-ray, MIMIC-CXR, and Chexpert Plus. The source code of this work is released on \url{https://github.com/Event-AHU/Medical_Image_Analysis}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑