Most current medical vision language models struggle to jointly generate diagnostic text and pixel-level segmentation masks in response to complex visual questions. This represents a major limitation towards clinical application, as assistive systems that fail to provide both modalities simultaneously offer limited value to medical practitioners. To alleviate this limitation, we first introduce RadDiagSeg-D, a dataset combining abnormality detection, diagnosis, and multi-target segmentation into a unified and hierarchical task. RadDiagSeg-D covers multiple imaging modalities and is precisely designed to support the development of models that produce descriptive text and corresponding segmentation masks in tandem. Subsequently, we leverage the dataset to propose a novel vision-language model, RadDiagSeg-M, capable of joint abnormality detection, diagnosis, and flexible segmentation. RadDiagSeg-M provides highly informative and clinically useful outputs, effectively addressing the need to enrich contextual information for assistive diagnosis. Finally, we benchmark RadDiagSeg-M and showcase its strong performance across all components involved in the task of multi-target text-and-mask generation, establishing a robust and competitive baseline.
Department of Medicine I, LMU University Hospital, LMU Munich, Germany（慕尼黑大学医学部第一部门，LMU大学医院，慕尼黑，德国）；Lunit Inc.（Lunit公司）
Large annotated datasets are essential for training robust Computer-Aided Diagnosis (CAD) models for breast cancer detection or risk prediction. However, acquiring such datasets with fine-detailed annotation is both costly and time-consuming. Vision-Language Models (VLMs), such as CLIP, which are pre-trained on large image-text pairs, offer a promising solution by enhancing robustness and data efficiency in medical imaging tasks. This paper introduces a novel Multi-View Mammography and Language Model for breast cancer classification and risk prediction, trained on a dataset of paired mammogram…
Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPA
Department of Diagnostic Imaging, Brown University Health（布朗大学健康中心诊断影像科）；Warren Alpert Medical School of Brown University（布朗大学沃伦·阿尔珀特医学院）；Department of Biomedical Engineering, Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院生物医学工程系）；Department of Radiology and Radiological Sciences, Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院放射科）；Johns Hopkins University Division of Pulmonary and Critical Care Medicine（约翰霍普金斯大学肺科与重症医学科）；Department of Radiology, University of Colorado School of Medicine（科罗拉多大学医学院放射科）
3D-RAD: A Comprehensive 3D Radiology Med-VQA Dataset with Multi-Temporal Analysis and Diverse Diagnostic Tasks
ZJU-Angelalign R&D Center for Intelligence Healthcare（浙江大学智能医疗研发中心）；Zhejiang University（浙江大学）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）；Guangdong Institute of Intelligence Science and Technology（广东省智能科学与技术研究院）
Radiology-GPT: A Large Language Model for Radiology
School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Electrical Engineering, Harvard University（哈佛大学工程与应用科学学院（电子工程系））；Department of Radiology, Second Xiangya Hospital（中南大学湘雅二医院放射科）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Department of Radiology, Massachusetts General Hospital and Harvard Medical School（麻省总医院及哈佛医学院放射科）；Department of Computer Science and Engineering, Lehigh University（利哈伊大学计算机科学与工程系）；Department of Computer Science and Engineering, University of Texas at Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Department of Radiation Oncology, Mayo Clinic（梅奥诊所放射肿瘤学系）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data
Northwestern Polytechnical University（西北工业大学）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）；Clinical Research Center for Medical Imaging in Hunan Province（湖南省医学影像临床研究中心）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Institute of Medical Research, Northwestern Polytechnical University（西北工业大学医学研究院）；Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；Central South University（中南大学）
Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation
New York University（纽约大学）；NYU Langone Health（纽约大学兰格医学中心）；Genentech（基因泰克）；NYU Grossman School of Medicine（纽约大学格罗斯曼医学院）；Wenzhou Medical University（温州医科大学）；Macau University of Science and Technology（澳门科学大学）
Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset
Stanford University（斯坦福大学）；Stanford Center for Artificial Intelligence in Medicine and Imaging（斯坦福大学医学与成像人工智能中心）；Department of Electrical Engineering（电气工程系）；University of California Berkeley（加州大学伯克利分校）；Department of Radiology（放射科）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Hospital Israelita Albert Einstein（以色列特医院阿尔伯特·爱因斯坦医院）
Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
Institute of Medical Research, Northwestern Polytechnical University（西北工业大学医学研究院）；School of Computing, University of Georgia（佐治亚大学计算学院）；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Lingang Laboratory（临港实验室）
Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology
Centre for Responsible Autonomous Systems in Healthcare (CRASH) Lab, Koita Centre for Digital Health（负责任的自主医疗系统中心（CRASH）实验室、Koita数字健康中心）；Ashoka University（阿什oka大学）；Independent Researcher（独立研究者）；Koita Centre for Digital Health（Koita数字健康中心）
College of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）；Thoracic Department, Fujian Medical University Union Hospital（福建医科大学附属协和医院胸外科）
Background and Objective:Computer-aided diagnosis (CAD) systems promote diagnosis effectiveness and alleviate pressure of radiologists. A CAD system for lung cancer diagnosis includes nodule candidate detection and nodule malignancy evaluation. Recently, deep learning-based pulmonary nodule detection has reached satisfactory performance ready for clinical application. However, deep learning-based nodule malignancy evaluation depends on heuristic inference from low-dose computed tomography volume to malignant probability, which lacks clinical cognition. Methods:In this paper, we propose a joint…
Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学）；Student Research Committee（学生研究委员会）；School of Allied Medical Science（Allied Medical Science学院）；Department of Health Information Technology and Management（卫生信息技术与管理系）
Purpose: Early detection and diagnosis of Covid-19 and accurate separation of patients with non-Covid-19 cases at the lowest cost and in the early stages of the disease are one of the main challenges in the epidemic of Covid-19. Concerning the novelty of the disease, the diagnostic methods based on radiological images suffer shortcomings despite their many uses in diagnostic centers. Accordingly, medical and computer researchers tended to use machine-learning models to analyze radiology images. Methods: Present systematic review was conducted by searching three databases of PubMed, Scopus, and…
Evaluating Multimodal LLMs for Inpatient Diagnosis: Real-World Performance, Safety, and Cost Across Ten Frontier Models
School of Computer Science and Applied Mathematics, University of the Witwatersrand（沃斯兰大学计算机科学与应用数学学院）；Wits MIND Institute, University of the Witwatersrand（沃斯兰大学Wits MIND研究所）；Grai Labs, Cape Town, South Africa（南非开普敦Grai实验室）；Faculty of Health Sciences, University of the Witwatersrand（沃斯兰大学健康科学学院）；South African Medical Research Council Vaccines and Infectious Diseases Analytics Research Unit, Faculty of Health Sciences, University of the Witwatersrand（南非医学研究委员会疫苗与传染病分析研究单位，沃斯兰大学健康科学学院）
LUMEN: Longitudinal Multi-Modal Radiology Model for Prognosis and Diagnosis
1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington DC, USA 2 Nvidia Corporation, Santa Clara, CA, USA 3 ETSI Telecomunicaci\' o n, Universidad Polit\' e cnica de Madrid, Madrid, Spain 4 School of Medicine；Health Sciences, George Washington University, Washington DC, USA
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Duke University（杜克大学）；The Chinese University of Hong Kong（香港中文大学）；Brigham and Women’s Hospital, Harvard Medical School（布莱根妇女医院，哈佛医学院）；Lenovo Research（联想研究院）；Lenovo Ltd.（联想集团有限公司）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Medical imaging plays a pivotal role in diagnosis and treatment in clinical practice. Inspired by the significant progress in automatic image captioning, various deep learning (DL)-based methods have been proposed to generate radiology reports for medical images. Despite promising results, previous works overlook the uncertainties of their models and are thus unable to provide clinicians with the reliability/confidence of the generated radiology reports to assist their decision-making. In this paper, we propose a novel method to explicitly quantify both the visual uncertainty and the textual u…
UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
Fung Institute for Engineering Leadership（工程领导力基金会）；University of California, Berkeley（加州大学伯克利分校）；Department of Radiology（放射科）；University of California, San Francisco（加州大学旧金山分校）；Division of Clinical Informatics and Digital Transformation（临床信息学与数字转型部）；Department of Neurological Surgery（神经外科部）
A Synthetic Data-Driven Radiology Foundation Model for Pan-tumor Clinical Diagnosis
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Lab（上海人工智能实验室）；The First Hospital of China Medical University（中国医科大学附属第一医院）；Harvard Medical School（哈佛医学院）
Radious: Unveiling the Enigma of Dental Radiology with BEIT Adaptor and Mask2Former in Semantic Segmentation
Gilan University of Medical Sciences（吉兰医科大学）；University of Tehran（德黑兰大学）；Hamedan University of Technology（哈马丹工业大学）；Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学）
Magnetic Resonance Imaging (MRI) is widely used for screening and staging prostate cancer. However, many prostate cancers have subtle features which are not easily identifiable on MRI, resulting in missed diagnoses and alarming variability in radiologist interpretation. Machine learning models have been developed in an effort to improve cancer identification, but current models localize cancer using MRI-derived features, while failing to consider the disease pathology characteristics observed on resected tissue. In this paper, we propose CorrSigNet, an automated two-step model that localizes p…
University of Central Florida（中佛罗里达大学）；National Institutes of Health（美国国立卫生研究院）
There are at least two categories of errors in radiology screening that can lead to suboptimal diagnostic decisions and interventions:(i)human fallibility and (ii)complexity of visual search. Computer aided diagnostic (CAD) tools are developed to help radiologists to compensate for some of these errors. However, despite their significant improvements over conventional screening strategies, most CAD systems do not go beyond their use as second opinion tools due to producing a high number of false positives, which human interpreters need to correct. In parallel with efforts in computerized analy…
University of Toronto（多伦多大学）；The Hospital for Sick Children (SickKids)（病童医院（SickKids））；Vector Institute（向量研究所）
Traditional datasets for the radiological diagnosis tend to only provide the radiology image alongside the radiology report. However, radiology reading as performed by radiologists is a complex process, and information such as the radiologist's eye-fixations over the course of the reading has the potential to be an invaluable data source to learn from. Nonetheless, the collection of such data is expensive and time-consuming. This leads to the question of whether such data is worth the investment to collect. This paper utilizes the recently published Eye-Gaze dataset to perform an exhaustive st…
Building RadiologyNET: Unsupervised annotation of a large-scale multimodal medical database
University of Rijeka – Faculty of Engineering（里耶卡大学工程学院）；Center for Artificial Intelligence and Cybersecurity（人工智能与网络安全中心）；Medical University of Graz（格拉茨医科大学）
We propose a data collecting and annotation pipeline that extracts information from Vietnamese radiology reports to provide accurate labels for chest X-ray (CXR) images. This can benefit Vietnamese radiologists and clinicians by annotating data that closely match their endemic diagnosis categories which may vary from country to country. To assess the efficacy of the proposed labeling technique, we built a CXR dataset containing 9,752 studies and evaluated our pipeline using a subset of this dataset. With an F1-score of at least 0.9923, the evaluation demonstrates that our labeling tool perform…
GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC)（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE)（医学影像、机器人、分析计算与学习中心）；Suzhou Institute for Advanced Research, USTC（苏州先进研究院，中国科学技术大学）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；T Magnetic Resonance Translational Medicine Research Center, Department of Radiology, The First Affiliated Hospital (Southwest Hospital) of Army Medical University（7T磁共振转化医学研究中心，放射科，中国医学大学第一附属医院（西南医院））
The University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；Shandong First Medical University（山东第一医科大学）；Shandong Academy of Medical Sciences（山东省医学科学院）；Shandong Cancer Hospital and Institute（山东省肿瘤医院）
There is an urgent need for automated methods to assist accurate and effective assessment of COVID-19. Radiology and nucleic acid test (NAT) are complementary COVID-19 diagnosis methods. In this paper, we present an end-to-end multitask learning (MTL) framework (COVID-MTL) that is capable of automated and simultaneous detection (against both radiology and NAT) and severity assessment of COVID-19. COVID-MTL learns different COVID-19 tasks in parallel through our novel random-weighted loss function, which assigns learning weights under Dirichlet distribution to prevent task dominance; our new 3D…
PanDent: Toward Comprehensive Tooth-Level Structure-Language Consistency in Dental Radiology
Faculty of Dentistry, The University of Hong Kong（香港大学牙医学院）；Imperial College London（帝国理工学院）；University of Science and Technology of China（中国科学技术大学）；Department of Data and Systems Engineering, The University of Hong Kong（香港大学数据与系统工程系）；Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）
A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT
Ain Shams University（艾因夏姆斯大学）；The Hong Kong University of Science and Technology（香港科技大学）；University of Catania（卡塔尼亚大学）；Universitat de Barcelona（巴塞罗那大学）
Radiology Report Generation Using Transformers Conditioned with Non-imaging Data
Centre for Computational Imaging and Simulation Technologies in Biomedicine（生物医学计算成像与模拟技术中心）；Leeds Institute for Cardiovascular and Metabolic Medicine（利兹心血管与代谢医学研究所）；School of Medicine（医学院）；School of Computing, University of Leeds（利兹大学计算机学院）
本文提出一种多模态 Transformer 网络，整合胸部 X 射线图像与患者人口统计信息生成放射报告，实验证明加入非影像数据有效提升了生成质量。
