3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models
Northwestern Polytechnical University(西北工业大学);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院);Clinical Research Center for Medical Imaging in Hunan Province(湖南省医学影像临床研究中心);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Georgia(佐治亚大学)
Hao Chen, Wei Zhao, Yingli Li, Tianyang Zhong, Yisong Wang, Youlan Shang, Lei Guo, Junwei Han, Tianming Liu, Jun Liu, Tuo Zhang
Medical image analysis is crucial in modern radiological diagnostics, especially given the exponential growth in medical imaging data. The demand for automated report generation systems has become increasingly urgent. While prior research has mainly focused on using machine learning and multimodal language models for 2D medical images, the generation of reports for 3D medical images has been less explored due to data scarcity and computational complexities. This paper introduces 3D-CT-GPT, a Visual Question Answering (VQA)-based medical visual language model specifically designed for generating radiology reports from 3D CT scans, particularly chest CTs. Extensive experiments on both public and private datasets demonstrate that 3D-CT-GPT significantly outperforms existing methods in terms of report accuracy and quality. Although current methods are few, including the partially open-source CT2Rep and the open-source M3D, we ensured fair comparison through appropriate data conversion and evaluation methodologies. Experimental results indicate that 3D-CT-GPT enhances diagnostic accuracy and report coherence, establishing itself as a robust solution for clinical radiology report generation. Future work will focus on expanding the dataset and further optimizing the model to enhance its performance and applicability.
Radiology-GPT: A Large Language Model for Radiology
School of Computing, University of Georgia(佐治亚大学计算机学院);Department of Electrical Engineering, Harvard University(哈佛大学工程与应用科学学院(电子工程系));Department of Radiology, Second Xiangya Hospital(中南大学湘雅二医院放射科);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);Department of Radiology, Massachusetts General Hospital and Harvard Medical School(麻省总医院及哈佛医学院放射科);Department of Computer Science and Engineering, Lehigh University(利哈伊大学计算机科学与工程系);Department of Computer Science and Engineering, University of Texas at Arlington(德克萨斯大学阿灵顿分校计算机科学与工程系);Department of Radiation Oncology, Mayo Clinic(梅奥诊所放射肿瘤学系);School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能医疗科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心)
MGH Radiology Llama: A Llama 3 70B Model for Radiology
School of Computing, University of Georgia(佐治亚大学计算学院);Department of Radiology, Massachusetts General Hospital and Harvard Medical School(麻省总医院及哈佛医学院放射科)
A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis
The University of Sydney(悉尼大学);Tencent AI Lab, Tencent(腾讯AI实验室);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Wollongong(伍伦贡大学);The University of Adelaide(阿德莱德大学)
MRScore: Evaluating Radiology Report Generation with LLM-based Reward System
The University of Sydney(悉尼大学);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Adelaide(阿德莱德大学);The University of Wollongong(卧龙岗大学)
Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports
National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆国家生物技术信息中心);National Institutes of Health Clinical Center(美国国立卫生研究院临床中心)
Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology
Centre for Responsible Autonomous Systems in Healthcare (CRASH) Lab, Koita Centre for Digital Health(负责任的自主医疗系统中心(CRASH)实验室、Koita数字健康中心);Ashoka University(阿什oka大学);Independent Researcher(独立研究者);Koita Centre for Digital Health(Koita数字健康中心)
RadioRAG: Online Retrieval-augmented Generation for Radiology Question Answering
University Hospital RWTH Aachen(亚琛工业大学附属医院);Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔朗根-纽伦堡大学);Stanford University(斯坦福大学);German Heart Centre Munich(慕尼黑德国心脏中心);Technical University of Munich(慕尼黑工业大学);Klinikum rechts der Isar(伊萨尔河右岸医院);TUD Dresden University of Technology(德累斯顿工业大学);National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心);University Hospital Heidelberg(海德堡大学附属医院)
Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
Institute of Medical Research, Northwestern Polytechnical University(西北工业大学医学研究院);School of Computing, University of Georgia(佐治亚大学计算学院);School of Life Science and Technology, University of Electronic Science and Technology of China(电子科技大学生命科学与技术学院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Electronics and Information, Northwestern Polytechnical University(西北工业大学电子信息学院);School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Lingang Laboratory(临港实验室)
Seeing Through Multiple Views: Parameter-Efficient Fine-Tuning via Selective Neurons for Consistent Radiology Report Generation
MedVisAI Lab(MedVisAI实验室);Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore(新加坡南洋理工大学李光前医学院);Centre of AI in Medicine, Singapore(新加坡人工智能医学中心);The Chinese University of Hong Kong, Sha Tin, New Territories, Hong Kong(香港中文大学);National Heart Centre Singapore (NHCS), Singapore(新加坡国家心脏中心)
Hallucination Filtering in Radiology Vision-Language Models Using Discrete Semantic Entropy
Lab for Artificial Intelligence in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(医学人工智能实验室,诊断与介入放射学部,RWTH亚琛大学医院);Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(诊断与介入放射学部,RWTH亚琛大学医院);Department of Diagnostic and Interventional Radiology, Technical University of Munich, School of Medicine and Health, Klinikum rechts der Isar, TUM University Hospital(诊断与介入放射学部,慕尼黑技术大学,医学院与健康学院,Klinikum rechts der Isar,TUM大学医院);Department of Cardiovascular Radiology and Nuclear Medicine, Technical University of Munich, School of Medicine and Health, German Heart Center, TUM University Hospital(心血管放射学与核医学部,慕尼黑技术大学,医学院与健康学院,德国心脏中心,TUM大学医院);Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(数字健康中心,医学院与卡尔·古斯塔夫·卡尔斯大学医院,德累斯顿技术大学);Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology(第一医学部,医学院与卡尔·古斯塔夫·卡尔斯大学医院,德累斯顿技术大学);Pathology & Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds(病理学与数据分析,圣詹姆斯医院医学研究所,利兹大学);Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg(医学肿瘤学,国家肿瘤疾病中心(NCT),海德堡大学医院)
ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data
Northwestern Polytechnical University(西北工业大学);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院);Clinical Research Center for Medical Imaging in Hunan Province(湖南省医学影像临床研究中心);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Institute of Medical Research, Northwestern Polytechnical University(西北工业大学医学研究院);Glasgow College, University of Electronic Science and Technology of China(电子科技大学格拉斯哥学院);Central South University(中南大学)
ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation
The University of Sydney(悉尼大学);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Adelaide(阿德莱德大学);The University of Wollongong(伍伦贡大学)
The Era of Foundation Models in Medical Imaging is Approaching : A Scoping Review of the Clinical Value of Large-Scale Generative AI Applications in Radiology
Development and Validation of a Large Language Model for Generating Fully-Structured Radiology Reports
Rensselaer Polytechnic Institute(伦斯勒理工学院);Massachusetts General Hospital, Harvard Medical School(哈佛医学院麻省总医院);Wake Forest University School of Medicine(维克森林大学医学院)
Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reports
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School(麻省总医院与哈佛医学院 Athinoula A. Martinos 生物医学成像中心);Charité - Universitätsmedizin Berlin(柏林夏里特医学院);Dana-Farber Cancer Institute(丹娜法伯癌症研究院)
Holistic Evaluation of GPT-4V for Biomedical Imaging
University of Georgia(佐治亚大学);Beijing Jiaotong University(北京交通大学);Northwestern Polytechnical University(西北工业大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);University of Electronic Science and Technology of China(电子科技大学)
Autonomous Structuring of Radiology Reports Across Modalities at Archive Scale Using an Open-Weight Large Language Model
Charité – Berlin University of Medicine(柏林夏里特医学院);Technical University of Munich(慕尼黑工业大学);University of Salerno(萨莱诺大学);Aristotle University of Thessaloniki(亚里士多德大学);Berlin University of Applied Sciences and Technology(柏林应用技术大学);Berlin Institute of Health(柏林健康研究院)
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo(东京大学医学研究生院放射学与生物医学工程系);Department of Radiology, The University of Tokyo Hospital(东京大学医院放射科);Department of Radiology, Kanazawa University Graduate School of Medical Sciences(金泽大学医学研究生院放射科);Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital(东京大学医院计算诊断放射学与预防医学系);Department of Radiology, School of Medicine, Jichi Medical University(自治医科大学医学院放射科)
Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration
Hatyaiwittayalai School(哈雅维泰莱学校);School of Information, Computer, and Communication Technology(信息、计算机与通信技术学院);Sirindhorn International Institute of Technology(Sirindhorn国际技术学院);Thammasat University(泰国朱拉大学);Pathum Thani, Thailand(帕亨他尼,泰国)
Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo(放射医学与生物医学工程系,东京大学医学研究生院);Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital(计算诊断放射学与预防医学系,东京大学医院);Department of Radiology, Kanazawa University Hospital(金泽大学医院放射科);Faculty of Medicine, The University of Tokyo(东京大学医学系);Department of Radiology, School of Medicine, Jichi Medical University(立命馆大学医学系放射科);Department of Radiology, The University of Tokyo Hospital(东京大学医院放射科)
Image and Data Mining in Reticular Chemistry Using GPT-4V
University of California, Berkeley(加州大学伯克利分校);Kavli Energy Nanoscience Institute(卡夫利能源纳米科学研究所);Bakar Institute of Digital Materials for the Planet(巴卡尔行星数字材料研究所);Stanford University(斯坦福大学);King Abdulaziz City for Science and Technology(阿卜杜勒阿齐兹国王科技城)
Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization
New York University(纽约大学);Columbia University Irving Medical Center(哥伦比亚大学欧文医学中心);NYU Grossman School of Medicine(纽约大学格罗斯曼医学院);University of California San Diego(加州大学圣地亚哥分校)