How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses
National Center for Biotechnology Information, National Library of Medicine(国家生物技术信息中心、美国国家医学图书馆);Clinical Center(临床中心);Imperial College London(伦敦帝国学院);King Abdullah University of Science & Technology(阿卜杜拉国王科技大学);Center for Information Technology, National Institutes of Health(美国国立卫生研究院信息技术中心)
Qingqing Zhu, Benjamin Hou, Tejas S. Mathai, Pritam Mukherjee, Qiao Jin, Xiuying Chen, Zhizheng Wang, Ruida Cheng, Ronald M. Summers, Zhiyong Lu
自动解读CT扫描能够减轻放射科医生的工作负担。然而,这一任务颇具挑战性,主要原因在于缺乏充足的数据集和评估参考标准。本研究旨在通过提出一种名为“GPTRadScore”的新型评估框架来弥补这一差距。该框架用于评估多模态大语言模型(例如带视觉功能的GPT-4(GPT-4V)、Gemini Pro Vision、LLaVA-Med和RadFM)为前瞻性识别的病灶生成描述的能力。通过采用基于GPT-4的分解技术,GPTRadScore将这些生成的描述与金标准报告语句进行对比,从身体部位、位置和病灶类型三个维度分析其准确性。评估结果显示,该框架与临床医生的评估具有高度相关性,并且展现出优于BLEU、METEOR和ROUGE等传统指标的潜力。此外,为助力未来研究,我们计划发布一个由临床医生标注的基准数据集。借助GPTRadScore,我们发现尽管GPT-4V和Gemini Pro Vision表现更佳,但它们的性能仍存在很大的提升空间,这主要是由于这些模型训练所用数据集的局限性所致。为验证这一潜力,我们对RadFM进行了微调,其准确性得到了显著提升:位置准确率从3.41%提升至12.8%,身体部位准确率从29.12%提升至53%,病灶类型准确率从9.24%提升至30%,从而验证了我们的假设。
英文摘要:
Automatically interpreting CT scans can ease the workload of radiologists. However, this is challenging mainly due to the scarcity of adequate datasets and reference standards for evaluation. This study aims to bridge this gap by introducing a novel evaluation framework, named ``GPTRadScore''. This framework assesses the capabilities of multi-modal LLMs, such as GPT-4 with Vision (GPT-4V), Gemini Pro Vision, LLaVA-Med, and RadFM, in generating descriptions for prospectively-identified findings. By employing a decomposition technique based on GPT-4, GPTRadScore compares these generated descriptions with gold-standard report sentences, analyzing their accuracy in terms of body part, location, and type of finding. Evaluations demonstrated a high correlation with clinician assessments and highlighted its potential over traditional metrics, such as BLEU, METEOR, and ROUGE. Furthermore, to contribute to future studies, we plan to release a benchmark dataset annotated by clinicians. Using GPTRadScore, we found that while GPT-4V and Gemini Pro Vision fare better, their performance revealed significant areas for improvement, primarily due to limitations in the dataset used for training these models. To demonstrate this potential, RadFM was fine-tuned and it resulted in significant accuracy improvements: location accuracy rose from 3.41\% to 12.8\%, body part accuracy from 29.12\% to 53\%, and type accuracy from 9.24\% to 30\%, thereby validating our hypothesis.
SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences(模式识别新实验室(NLPR)、多模态人工智能系统国家重点实验室(MAIS)、自动化研究所、中国科学院);Cuiying Honors College, Lanzhou University(cuiying荣誉学院、兰州大学);Research Institute of Intelligent Complex Systems, Fudan University(智能复杂系统研究院、复旦大学);Zhongguancun Academy(中关村学院)
Automated segmentation and extraction of posterior eye segment using OCT scans
School of Automation Science and Electrical Engineering, Beihang University (BUAA)(北京航空航天大学自动化科学与电气工程学院);Khalifa University of Science & Technology(哈利法科技大学);School of Computer and Communication Engineering, University of Science & Technology Beijing(北京科技大学计算机与通信工程学院);School of Electrical Engineering and Intelligentization, Dongguan University of Technology(东莞理工学院电气工程与智能化学院)
Automated Plaque Detection and Agatston Score Estimation on Non-Contrast CT Scans: A Multicenter Study
National Institutes of Health Clinical Center(美国国立卫生研究院临床中心);National Institute of Arthritis and Musculoskeletal and Skin Diseases(美国国立关节炎、肌肉骨骼和皮肤病研究所)
Automatic Pulmonary Nodule Detection in CT Scans Using Convolutional Neural Networks Based on Maximum Intensity Projection
University of Groningen, University Medical Center Groningen(格罗宁根大学医学中心);Tianjin Medical University Cancer Institute and Hospital(天津医科大学肿瘤医院);University of Twente(特温特大学)
Automated Radiological Report Generation For Chest X-Rays With Weakly-Supervised End-to-End Deep Learning
Nanjing University(南京大学);Drum Tower Hospital, Nanjing University(南京大学医学院附属鼓楼医院);Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院北京协和医学院)
Automatic detection and diagnosis of sacroiliitis in CT scans as incidental findings
The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Hadassah Hebrew University Medical Center(哈达萨希伯来大学医学中心);Sheba Medical Center(舍巴医学中心);Tel Aviv University(特拉维夫大学)
Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM)
School of Biomedical Engineering, King’s College London(伦敦国王学院生物医学工程学院);King’s College Hospital(国王学院医院);Institute of Psychiatry, Psychology & Neuroscience, King’s College London(伦敦国王学院精神病学、心理学与神经科学研究所);Centre for Medical Image Computing, Dementia Research, University College London(伦敦大学学院医学影像计算中心(痴呆研究))
Automated triaging of head MRI examinations using convolutional neural networks
King’s College London(伦敦国王学院);King’s College Hospital(国王学院医院);Wrightington, Wigan and Leigh NHSFT(赖廷顿、维冈与利国家医疗服务基金会信托);University College London(伦敦大学学院)
Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset
University of Oklahoma(俄克拉荷马大学);University of Memphis(孟菲斯大学);Marquette University(马凯特大学)
Autonomous Navigation of an Ultrasound Probe Towards Standard Scan Planes with Deep Reinforcement Learning
The Chinese University of Hong Kong(香港中文大学);Shenzhen University(深圳大学);Sonoscape Medical Corp.(开立医疗);Peking University Shenzhen Hospital(北京大学深圳医院);Southern University of Science and Technology(南方科技大学);Shenzhen Research Institute of the Chinese University of Hong Kong(香港中文大学深圳研究院)