In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM. We consider the construction of foundational models from three perspectives, namely, dataset construction, model design, and thorough evaluation. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, which consists of 16M 2D and 3D medical scans with high-quality text descriptions or reports across various data formats, modalities, and tasks, covering over 5000 distinct diseases. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans; (ii), we propose an architecture that enables visually conditioned generative pre-training, i.e., allowing for integration of text input with 2D or 3D medical scans, and generate responses for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently fine-tuned on the domain-specific dataset, which is a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs, termed as RadMD; (iii), we propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. We conduct both automatic and human evaluation on RadBench, in both cases, RadFM outperforms existing multi-modal foundation models, that are publicaly accessible, including Openflamingo, MedFlamingo, MedVInT and GPT-4V. Additionally, we also adapt RadFM for different public benchmarks, surpassing existing SOTAs on diverse datasets. All codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.
Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiology
Shanghai Jiao Tong University（上海交通大学）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Department of Oral Craniomaxillofacial（上海第九人民医院，上海交通大学医学院，口腔颅面医学部）；Shanghai Jiao Tong University, School of Computer Science（上海交通大学，计算机科学学院）；University of Electronic Science and Technology of China, School of Mechanical and Electrical Engineering（电子科技大学，机械与电子工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University, College of Stomatology（上海交通大学，口腔医学院）；National Center for Stomatology（国家口腔医学中心）；National Clinical Medical Research Center for Oral Diseases（国家口腔疾病临床医学研究中心）；Shanghai Key Laboratory of Stomatology（上海口腔医学重点实验室）；Shanghai Research Institute of Stomatology（上海口腔研究所）；Chinese Academy of Medical Science, Research Unit of Oral and Maxillofacial Regenerative Medicine（中国医学科学院，口腔颌面再生医学研究单元）
BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
Lehigh University（理海大学）；University of Georgia（佐治亚大学）；Samsung Research America（三星美国研究院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Pennsylvania（宾夕法尼亚大学）；Children’s Hospital of Philadelphia（费城儿童医院）；University of Central Florida（中佛罗里达大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；UTHealth Houston（休斯顿德克萨斯大学健康科学中心）；Mayo Clinic（妙佑医疗国际）；Penn Institute for Biomedical Informatics（宾夕法尼亚生物医学信息学研究所）；Leonard Davis Institute of Health Economics（伦纳德·戴维斯健康经济学研究所）；Stanford University School of Medicine（斯坦福大学医学院）；Stanford University（斯坦福大学）
Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation
Sun Yat-Sen Memorial Hospital（孙逸仙纪念医院）；Sun Yat-Sen University（中山大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；Xunfei Healthcare Technology Co., Ltd.（讯飞医疗科技股份有限公司）
OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation
Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉A.马丁诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；University Campus Bio-Medico of Rome（罗马生物医学大学校园）
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Wake Forest University School of Medicine（维克森林大学医学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Health system learning achieves generalist neuroimaging models
Machine Learning in Neurosurgery Lab University of Michigan（密歇根大学机器学习神经外科实验室）；University of Michigan Computational Medicine and Bioinformatics（密歇根大学计算医学与生物信息学）；University of Michigan Computer Science and Engineering（密歇根大学计算机科学与工程）；University of Michigan Radiology（密歇根大学放射学）
MAIRA-1: A specialised large multimodal model for radiology report generation
Health Futures, Microsoft Research（微软研究院健康未来部门）；Microsoft Research India（微软研究院印度分部）；Microsoft Azure AI（微软Azure人工智能）；Microsoft Health and Life Sciences（微软健康与生命科学部门）
A Generalist Foundation Model for Total-body PET/CT Enables Diagnostic Reporting and System-wide Metabolic Profiling
Shandong First Medical University & Shandong Academy of Medical Sciences（山东第一医科大学（山东省医学科学院））；Shandong Provincial Hospital, Shandong First Medical University（山东第一医科大学附属省立医院）；Hubei Cancer Hospital, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属湖北肿瘤医院）；The Second Affiliated Hospital of Shandong First Medical University（山东第一医科大学第二附属医院）；The first Affiliated Hospital of Shandong First Medical University（山东第一医科大学第一附属医院）
Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
Department of Quantitative Biomedicine, University of Zurich（苏黎世大学定量生物医学系）；ETH AI Center, ETH Zurich（苏黎世联邦理工学院人工智能中心）；International School of Medicine, Istanbul Medipol University（伊斯坦布尔梅迪波尔大学国际医学院）；Department of Electrical and Computer Engineering, Boston University（波士顿大学电气与计算机工程系）；Division of Intramural Research, National Institutes of Health（美国国立卫生研究院内部研究部）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；Institute for Diagnostic and Interventional Radiology, University Hospital Zurich（苏黎世大学医院诊断与介入放射学研究所）；Department Artificial Intelligence in Biomedical Engineering, FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡工业大学生物医学工程人工智能系）
Microsoft Health Futures UK（微软健康未来英国）；Microsoft Health Futures US（微软健康未来美国）；Mayo Clinic M.D（梅奥诊所）；Radiology AI Lab, Mayo Clinic M.D（放射学AI实验室，梅奥诊所）；Microsoft Health & Life Sciences（微软健康与生命科学）；Cambridge University Hospitals M.D（剑桥大学医院）
Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医科大学）；Student Research Committee（学生研究委员会）；School of Allied Medical Science（Allied Medical Science学院）；Department of Health Information Technology and Management（卫生信息技术与管理系）
Purpose: Early detection and diagnosis of Covid-19 and accurate separation of patients with non-Covid-19 cases at the lowest cost and in the early stages of the disease are one of the main challenges in the epidemic of Covid-19. Concerning the novelty of the disease, the diagnostic methods based on radiological images suffer shortcomings despite their many uses in diagnostic centers. Accordingly, medical and computer researchers tended to use machine-learning models to analyze radiology images. Methods: Present systematic review was conducted by searching three databases of PubMed, Scopus, and…
Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
University of Naples Federico II, Department of Electrical Engineering and Information Technology（那不勒斯费德里科二世大学电子工程与信息技术系）；Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science（西北大学计算机科学系，工程与应用科学学院）；Pegaso University, Department of Information Science and Technology（佩加索大学信息科学与技术系）
Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（计算机科学与工程系，香港科学与技术大学）
Examining Deployment and Refinement of the VIOLA-AI Intracranial Hemorrhage Model Using an Interactive NeoMedSys Platform
Computational Radiology & Artificial Intelligence Unit, Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine（计算放射学与人工智能单元，物理与计算放射学部门，放射科与核医学诊所）；Oslo University Hospital（奥斯陆大学医院）；Department of Physics（物理部门）；University of Oslo（奥斯陆大学）；Department of Medical Biophysics（医学生物物理学部门）；University of Toronto（多伦多大学）
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
PleIAs；Sorbonne Center for Artificial Intelligence（索邦人工智能中心）；Sciences Po Médialab（巴黎政治学院媒体实验室）；EPFL（瑞士洛桑联邦理工学院）；CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学CAIRO）；Paris Dauphine-PSL（巴黎多芬纳-PSL大学）；Lattice, ENS-PSL（巴黎高等师范学院-巴黎文理研究大学Lattice实验室）；TU Munich, Munich Center for Machine Learning（慕尼黑工业大学慕尼黑机器学习中心）；KU Leuven（鲁汶大学）
Center for Artificial Intelligence in Medicine and Imaging（医学与成像人工智能中心）；Stanford University（斯坦福大学）
Multi-modal foundation models are typically trained on millions of pairs of natural images and text captions, frequently obtained through web-crawling approaches. Although such models depict excellent generative capabilities, they do not typically generalize well to specific domains such as medical images that have fundamentally shifted distributions compared to natural images. Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets. Thus, in this study, we seek to research and expand the representational capab…
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
A generalizable large-scale foundation model for musculoskeletal radiographs
Seoul National University Hospital（首尔大学医院）；Seoul National University（首尔大学）；Seoul National University College of Medicine（首尔大学医学院）；Seoul National University Bundang Hospital（首尔大学盆唐医院）
MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
Centre for Perceptual and Interactive Intelligence, the Chinese University of Hong Kong（香港中文大学感知与交互智能中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Department of Electronic Engineering, the Chinese University of Hong Kong（香港中文大学电子工程系）
MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images
School of Biomedical Engineering & Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）；NVIDIA（英伟达）；Department of Medical Physics and Biomedical Engineering, University College London（伦敦大学学院医学物理与生物医学工程系）
MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Models
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University（IDLab，根特大学）；VERSES AI Research Lab（VERSES AI研究实验室）；Mila, University of Montreal（米拉研究所，蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）
