The latest breakthroughs in large vision-language models, such as Bard and GPT-4, have showcased extraordinary abilities in performing a wide range of tasks. Such models are trained on massive datasets comprising billions of public image-text pairs with diverse tasks. However, their performance on task-specific domains, such as radiology, is still under-investigated and potentially limited due to a lack of sophistication in understanding biomedical images. On the other hand, conversational medical models have exhibited remarkable success but have mainly focused on text-based analysis. In this paper, we introduce XrayGPT, a novel conversational medical vision-language model that can analyze and answer open-ended questions about chest radiographs. Specifically, we align both medical visual encoder (MedClip) with a fine-tuned large language model (Vicuna), using a simple linear transformation. This alignment enables our model to possess exceptional visual conversation abilities, grounded in a deep understanding of radiographs and medical domain knowledge. To enhance the performance of LLMs in the medical context, we generate ~217k interactive and high-quality summaries from free-text radiology reports. These summaries serve to enhance the performance of LLMs through the fine-tuning process. Our approach opens up new avenues the research for advancing the automated analysis of chest radiographs. Our open-source demos, models, and instruction sets are available at: https://github.com/mbzuai-oryx/XrayGPT.
Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
University College Dublin(都柏林大学学院);School of Computer Science, University College Dublin(都柏林大学学院计算机科学学院);The Third Affiliated Hospital of Southern Medical University(南方医科大学第三附属医院);Dublin City University(都柏林城市大学)
Vision-language models for chest radiography do not always need the image
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg(弗里德里希-亚历山大-埃尔朗根-纽伦堡大学模式识别实验室);Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich(慕尼黑工业大学医学院与健康学院伊萨尔河右岸医院诊断与介入放射学系);Lab for AI in Medicine, RWTH Aachen University(亚琛工业大学医学人工智能实验室);Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(亚琛工业大学医院诊断与介入放射学系)
Automated diagnosis of pneumothorax using an ensemble of convolutional neural networks with multi-sized chest radiography images
KAIST(韩国科学技术院)
气胸是一种相对常见的疾病,但在某些情况下,它可能很难发现胸部X线摄影。在本文中,我们提出了一种在胸部X光检查中检测气胸的新方法。我们提出了一个集成模型的相同的卷积神经网络(CNN)与三个不同大小的射线照相图像。传统的方法可能无法正确地表征丢失的特征,同时将大尺寸图像重新定位为256 x 256或224 x 224尺寸。我们的模型与胸部X射线数据集,其中包含超过100,000个胸部X射线摄影图像进行了评估。作为实验的结果,所提出的模型显示AUC 0.911,这是气胸检测的最新结果。我们的方法预计将是有效的应用时,CNN大尺寸的医学图像。
Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
Massachusetts Institute of Technology(麻省理工学院);Philips Research North America(飞利浦北美研究院);Beth Israel Deaconess Medical Center, Harvard Medical School(哈佛医学院贝斯以色列女执事医疗中心)
An Ordinal Regression Framework for a Deep Learning Based Severity Assessment for Chest Radiographs
RWTH Aachen University Hospital(亚琛工业大学医院);Visual Computing Institute, RWTH Aachen University(亚琛工业大学视觉计算研究所);Institute of Medical Informatics, RWTH Aachen University Hospital(亚琛工业大学医院医学信息学研究所);Department of Oral and Maxillofacial Surgery, RWTH Aachen University Hospital(亚琛工业大学医院口腔颌面外科)
A Generative Foundation Model for Chest Radiography
Department of Radiation Oncology, Stanford University School of Medicine(斯坦福大学医学院放射肿瘤科);Department of Computer Science, School of Computing and Data Science, The University of Hong Kong(香港大学计算机科学系);Department of Population Medicine and Diagnostic Sciences, College of Veterinary Medicine, Cornell University(康奈尔大学兽医学院人口医学与诊断科学系);Department of Radiology, The First Affiliated Hospital of Jinan University(暨南大学第一附属医院放射科);Department of Radiology, Stanford University School of Medicine(斯坦福大学医学院放射科);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系)
CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University(斯坦福大学医学与影像人工智能中心);Department of Radiology, Stanford University(斯坦福大学放射学系)
Automated Estimation of Total Lung Volume using Chest Radiographs and Deep Learning
Radboud university medical center(拉德堡德大学医学中心)
肺总容积是一种重要的定量生物标志物,用于评估限制性肺部疾病。在这项研究中,我们研究了几种深度学习方法的性能,用于自动测量胸片中的总肺容量。从有胸部CT的患者中收集了7621张后前位和侧位胸片(CXR)。同样,从具有肺功能检查(PFT)结果的患者中选择了928项CXR研究。分别根据CT或PFT数据上的肺分割计算参考总肺体积。该数据集用于训练深度学习架构,以根据胸片预测总肺容量。实验以逐步增加复杂性的方式构建,以证明仅使用CT衍生标签进行训练的效果和错误来源。最优模型在291例CXR研究中进行了测试,参考肺体积从PFT获得。最佳深度学习回归模型显示,使用正面和侧面胸片作为输入,MAE为408 ml,MAPE为8.1%,Pearson r = 0.92。CT衍生的标签对于预训练是有用的,但是通过用PFT衍生的标签微调网络来获得最佳性能。我们首次证明,最先进的深度学习解决方案可以从普通胸片中准确测量总肺容量。所提出的模型可用于获得总肺体积从常规采集的胸片在没有额外的成本,并可能是一个有用的工具,以确定随着时间的推移,定期提交胸部X射线的患者的趋势。
Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformers
Imperial College London(帝国理工学院);Causality in Healthcare AI Hub(医疗AI因果关系中心);University of Edinburgh(爱丁堡大学);Cleveland Clinic London(克利夫兰诊所伦敦);Department of Perioperative Medicine, CHU Clermont-Ferrand(克莱蒙费朗大学医院围手术期医学科);Department of Medicine, Massachusetts General Hospital(麻省总医院医学部);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所)
Deep Learning to Quantify Pulmonary Edema in Chest Radiographs
Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Harvard Medical School(哈佛医学院);Massachusetts Institute of Technology(麻省理工学院);Philips Research(飞利浦研究院)
Bone Suppression on Chest Radiographs With Adversarial Learning
Imaging Biomarkers and Computer-aided Diagnosis Laboratory, Radiology and Imaging Sciences, National Institutes of Health Clinical Center(美国国立卫生研究院临床中心放射学与成像科学部成像生物标志物和计算机辅助诊断实验室);Ping An Technology Co., Ltd.(平安科技有限公司)
HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London(生物统计学与健康信息学系,精神病学、心理学与神经科学研究所,伦敦国王学院)
Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation
Centre for Perceptual and Interactive Intelligence(感知与交互智能中心);the Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Sensetime Research(商汤科技研究院);Department of Electronic Engineering(电子工程系)
Deformable Image Registration of Dark-Field Chest Radiographs for Local Lung Signal Change Assessment
Technical University of Munich(慕尼黑工业大学);Munich Institute for Biomedical Engineering, Technical University of Munich(慕尼黑工业大学慕尼黑生物医学工程研究所);Klinikum rechts der Isar, Technical University of Munich(慕尼黑工业大学右岸医院);TUM University Hospital(慕尼黑工业大学大学医院);Philips Innovative Technologies(飞利浦创新技术公司);Institute for Advanced Study, Technical University of Munich(慕尼黑工业大学高等研究院)
A Generic Approach to Lung Field Segmentation from Chest Radiographs using Deep Space and Shape Learning
Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Health System(儿童国家卫生系统谢赫扎耶德儿科外科创新研究所);Children’s National Health System(儿童国家卫生系统);San Francisco State University(旧金山州立大学);School of Medicine and Health Sciences, George Washington University(乔治华盛顿大学医学与健康科学学院)
已提出将计算机辅助诊断(CAD)技术用于成人队列的胸片(CXR)肺野分割,但很少用于儿科受试者。统计形状模型(SSM),最先进的基于CXR的肺野分割方法的主力,不能有效地适应在儿科发育阶段的肺野的形状变化。我们工作的主要贡献是:(1)来自CXR的通用肺野分割框架,适应成人和儿科队列的大形状变化;(2)深度表示学习检测机制\emph{ensemble space learning},用于鲁棒的对象定位;以及(3)\emph{marginal shape deep learning},用于形状变形参数估计。与传统SSM的迭代方法不同,所提出的形状学习机制将参数空间转换为边缘子空间,这些边缘子空间使用递归表示学习机制有效地求解。此外,我们的方法是第一个在基于CXR的肺分割中包括具有挑战性的心脏后区域,以进行准确的肺活量估计。该框架在3个月至89岁的患者的668个CXR上进行评估。我们得到平均Dice相似系数为$0.96\pm0.03$(包括心脏后区域)。对于一个给定的精度,所提出的方法也被发现比传统的基于SSM的迭代分割方法更快。所提出的通用框架的计算简单性可以类似地应用于其他可变形对象的快速分割。