Comprehensive language-image pre-training for 3D medical image understanding
Microsoft（微软公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Department of Radiology, University of Cambridge and Cambridge University Hospitals NHS Foundation Trust（剑桥大学放射科及剑桥大学医院国家健康服务体系基金会信托）；Pattern Analysis and Learning Group, Heidelberg University Hospital（海德堡大学医院模式分析与学习组）；Department of Radiology, Mayo Clinic（梅奥诊所放射科）
Tassilo Wald, Ibrahim Ethem Hamamci, Yuan Gao, Sam Bond-Taylor, Harshita Sharma, Maximilian Ilse, Cynthia Lo, Olesya Melnichenko, Anton Schwaighofer, Noel C. F.… 展开作者
Tassilo Wald, Ibrahim Ethem Hamamci, Yuan Gao, Sam Bond-Taylor, Harshita Sharma, Maximilian Ilse, Cynthia Lo, Olesya Melnichenko, Anton Schwaighofer, Noel C. F. Codella, Maria Teodora Wetscherek, Klaus H. Maier-Hein, Panagiotis Korfiatis, Valentina Salvatelli, Javier Alvarez-Valle, Fernando Pérez-García
In the 3D medical image domain, vision-language pre-training is used to create vision-language encoders (VLEs) that can support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, three challenges limit the capabilities of current 3D VLEs: data scarcity due to privacy concerns, high computational costs resulting from the volumetric nature of the images, and a domain shift between the long reports used for training and the short prompts used during inference for, e.g., zero-shot classification. As a consequence, natural-image VLE recipes do not directly transfer to 3D medical imaging.
In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training, allowing us to leverage both image-only and paired image-text 3D datasets. Further, we propose a novel loss that addresses the domain shift between long reports and short textual prompts. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification.
The model weights and inference code are freely available at https://huggingface.co/microsoft/colipri.
Osaka Metropolitan University（大阪公立大学）；Smart Life Science Lab, Center for Health Science Innovation, Osaka Metropolitan University（大阪公立大学健康科学创新中心智能生命科学实验室）
The goal of our research is to create a comprehensive and flexible library that is easy to use for medical imaging research, and capable of handling grayscale images, multiple inputs (both images and tabular data), and multi-label tasks. We have named it Nervus. Based on the PyTorch library, which is suitable for AI for research purposes, we created a four-part model to handle comprehensive inputs and outputs. Nervus consists of four parts. First is the dataloader, then the feature extractor, the feature mixer, and finally the classifier. The dataloader preprocesses the input data, the feature…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Cincinnati Children’s Hospital Medical Center（辛辛那提儿童医院医疗中心）；University of Cincinnati College of Medicine（辛辛那提大学医学院）；University of Cincinnati（辛辛那提大学）
Image registration is a critical component in the applications of various medical image analyses. In recent years, there has been a tremendous surge in the development of deep learning (DL)-based medical image registration models. This paper provides a comprehensive review of medical image registration. Firstly, a discussion is provided for supervised registration categories, for example, fully supervised, dual supervised, and weakly supervised registration. Next, similarity-based as well as generative adversarial network (GAN)-based registration are presented as part of unsupervised registrat…
Federated Learning for Medical Image Classification: A Comprehensive Benchmark
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（华南理工大学吴贤铭智能工程学院）
A comprehensive survey on deep active learning in medical image analysis
Fudan University（复旦大学）；School of Basic Medical Sciences, Fudan University（复旦大学基础医学院）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
In the realm of medical imaging, the training of machine learning models necessitates a large and varied training dataset to ensure robustness and interoperability. However, acquiring such diverse and heterogeneous data can be difficult due to the need for expert labeling of each image and privacy concerns associated with medical data. To circumvent these challenges, data augmentation has emerged as a promising and cost-effective technique for increasing the size and diversity of the training dataset. In this study, we provide a comprehensive review of the specific data augmentation techniques…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；University of Cincinnati College of Medicine（辛辛那提大学医学院）；University of Cincinnati（辛辛那提大学）
The outbreak of novel coronavirus disease (COVID- 19) has claimed millions of lives and has affected all aspects of human life. This paper focuses on the application of deep learning (DL) models to medical imaging and drug discovery for managing COVID-19 disease. In this article, we detail various medical imaging-based studies such as X-rays and computed tomography (CT) images along with DL methods for classifying COVID-19 affected versus pneumonia. The applications of DL techniques to medical images are further described in terms of image localization, segmentation, registration, and classifi…
Deep Learning Approaches for Medical Imaging Under Varying Degrees of Label Availability: A Comprehensive Survey
University College Dublin（都柏林大学学院）；North China Institute of Aerospace Engineering（北华航天工业学院）；Hithink RoyalFlush Information Network Co., Ltd（同花顺信息网络股份有限公司）
How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model
Department of Electrical and Computer Engineering, Duke University, US（杜克大学电气与计算机工程系）；Department of Biostatistics & Bioinformatics, Duke University, US（杜克大学生物统计学与生物信息学系）；Department of Computer Science, Duke University, US（杜克大学计算机科学系）；Department of Radiology, Duke University, US（杜克大学放射学系）
A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond
Indian Institute of Technology Indore, India；R.M.D. Engineering College, Kavaraipettai, India；Jio Platforms Limited, India；Birla Institute of Technology \& Science Pilani, India
Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision
South Dakota State University（南达科他州立大学）；RWTH Aachen University（亚琛工业大学）；University of Kentucky（肯塔基大学）；University of Regensburg（雷根斯堡大学）；Iran University of Science and Technology（伊朗科技大学）；Imperial College London（伦敦帝国理工学院）
De-Identification of Medical Imaging Data: A Comprehensive Tool for Ensuring Patient Privacy
Institute for AI in Medicine (IKIM)（人工智能医学研究所（IKIM））；University Hospital Essen（埃森大学医院）；Cancer Research Center Cologne Essen (CCCE)（科隆-埃森癌症研究中心（CCCE））；University Medicine Essen（埃森大学医学部）；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））；Technical University Dortmund（多特蒙德工业大学）
Federated Learning for Large Models in Medical Imaging: A Comprehensive Review
School of Cyber Science and Engineering, Sichuan University（四川大学网络科学与工程学院）；Sichuan Institute of Computer Sciences（四川省计算机科学研究院）；College of Computer Science, Sichuan University（四川大学计算机学院）；Department of Mathematics, The Chinese University of Hong Kong（香港中文大学数学系）；School of Electrical and Electronic Engineering, College of Engineering, Yonsei University（延世大学工程学院电子与电气工程系）
Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review
University of Regensburg（雷根斯堡大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所）；RWTH Aachen University（亚琛工业大学）；Iran University of Science and Technology（伊朗科技大学）；Shahid Beheshti University（沙希德·贝赫什提大学）
ETH Zurich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University Hospital of Tübingen（蒂宾根大学医院）；Kantonsspital Winterthur（温特图尔州立医院）；GlaxoSmithKline（葛兰素史克）
Privacy concerns around sharing personally identifiable information are a major practical barrier to data sharing in medical research. However, in many cases, researchers have no interest in a particular individual's information but rather aim to derive insights at the level of cohorts. Here, we utilize Generative Adversarial Networks (GANs) to create derived medical imaging datasets consisting entirely of synthetic patient data. The synthetic images ideally have, in aggregate, similar statistical properties to those of a source dataset but do not contain sensitive personal information. We ass…
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）；University Hospitals Leuven（鲁汶大学医院）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Accurate medical image segmentation is essential for diagnosis and treatment planning of diseases. Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they are still challenged by complicated conditions where the segmentation target has large variations of position, shape and scale, and existing CNNs have a poor explainability that limits their application to clinical decisions. In this work, we make extensive use of multiple attentions in a CNN architecture and propose a comprehensive attention-based CNN (CA-Net) f…
Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects
Faculty of Informatics and Data Science, University of Regensburg（雷根斯堡大学信息学与数据科学学院）；University of California, Davis（加州大学戴维斯分校）；Motilal Nehru National Institute of Technology Allahabad（莫蒂拉尔·尼赫鲁国立阿拉哈巴德技术学院）；Faculty of Electrical Engineering and Information Technology, RWTH Aachen University（亚琛工业大学电气工程与信息技术学院）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）
National Heart and Lung Institute, Faculty of Medicine, Imperial College London（帝国理工学院医学院国家心肺研究所）；Department of Bioengineering, Faculty of Engineering, Imperial College London（帝国理工学院工程系生物工程系）；Royal Brompton Hospital（皇家布朗普顿医院）
Various structures in human physiology follow a treelike morphology, which often expresses complexity at very fine scales. Examples of such structures are intrathoracic airways, retinal blood vessels, and hepatic blood vessels. Large collections of 2D and 3D images have been made available by medical imaging modalities such as magnetic resonance imaging (MRI), computed tomography (CT), Optical coherence tomography (OCT) and ultrasound in which the spatial arrangement can be observed. Segmentation of these structures in medical imaging is of great importance since the analysis of the structure …
Modality-Projection Universal Model for Comprehensive Full-Body Medical Imaging Segmentation
the Institute of Medical Technology and National Biomedical Imaging Center, Peking University（北京大学医学技术研究所暨国家生物医学成像中心）；The First Affiliated Hospital of Shandong First Medical University & Shandong Provincial Qianfoshan Hospital（山东第一医科大学第一附属医院暨山东省千佛山医院）；Peking University Third Hospital（北京大学第三医院）；Guangdong Provincial People’s Hospital（广东省人民医院）；Peking University Cancer Hospital & Institute（北京大学肿瘤医院暨肿瘤研究所）
Rutgers University（罗格斯大学）；SenseBrain Research（SenseBrain研究院）；National Heart Centre Singapore（新加坡国家心脏中心）；Duke-National University Of Singapore（杜克-新加坡国立大学）
Multi-modality images have been widely used and provide comprehensive information for medical image analysis. However, acquiring all modalities among all institutes is costly and often impossible in clinical settings. To leverage more comprehensive multi-modality information, we propose a privacy secured decentralized multi-modality adaptive learning architecture named ModalityBank. Our method could learn a set of effective domain-specific modulation parameters plugged into a common domain-agnostic network. We demonstrate by switching different sets of configurations, the generator could outpu…
Super-Generalist: Towards Comprehensive and Accurate Medical Image Understanding via Generalist-Specialist Synergy
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Lab（湖畔实验室）；Department of Radiology, Ningbo No. 2 Hospital（宁波市第二医院放射科）；Department of Radiology, Guangdong Provincial People’s Hospital（广东省人民医院放射科）；Department of Radiology, Shanghai Institution of Pancreatic Disease（上海胰腺疾病研究所放射科）；Department of Radiology, Shengjing Hospital of China Medical University（中国医科大学附属盛京医院放射科）；The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第一医院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Benchmarking Real-World Medical Image Classification with Noisy Labels: Challenges, Practice, and Outlook
Japan Advanced Institute of Science and Technology（日本科学技术先进研究院）；University College London（伦敦大学学院）；The Hong Kong University of Science and Technology（香港科学大学）；University of Toyama（toyama大学）；Monash University（墨尔本大学）
The Hong Kong University of Science and Technology（香港科技大学）
Affine registration is indispensable in a comprehensive medical image registration pipeline. However, only a few studies focus on fast and robust affine registration algorithms. Most of these studies utilize convolutional neural networks (CNNs) to learn joint affine and non-parametric registration, while the standalone performance of the affine subnetwork is less explored. Moreover, existing CNN-based affine registration approaches focus either on the local misalignment or the global orientation and position of the input to predict the affine transformation matrix, which are sensitive to spati…
Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis
School of Biomedical Engineering（生物医学工程学院）；University of British Columbia（不列颠哥伦比亚大学）；Tarbiat Modares University（塔比亚特·莫达雷斯大学）；University of Regensburg（雷根斯堡大学）；South Dakota State University（南达科他州立大学）；RWTH Aachen University（亚琛工业大学）；Lunit Inc.（Lunit公司）；Northwestern University（西北大学）；Fraunhofer Institute for Digital Medicine（弗劳恩霍夫数字医学研究院）
Kaapana: A Comprehensive Open-Source Platform for Integrating AI in Medical Imaging Research Environments
Division of Medical Image Computing, German Cancer Research Center, Heidelberg, Germany（德国癌症研究中心医学影像计算部，海德堡，德国）；Medical Faculty, University of Heidelberg, Heidelberg, Germany（海德堡大学医学系，海德堡，德国）；Computer Science Faculty, University of Heidelberg, Heidelberg, Germany（海德堡大学计算机科学系，海德堡，德国）；Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany（放射肿瘤学部门模式分析与学习组，海德堡大学医院，海德堡，德国）；German Cancer Consortium (DKTK), Partner Site Heidelberg, Heidelberg, Germany（德国癌症联盟（DKTK），海德堡合作伙伴站点，海德堡，德国）；Heidelberg Institute of Radiation Oncology (HIRO), National Center for Radiation Research in Oncology (NCRO), Heidelberg, Germany（海德堡放射肿瘤学研究所（HIRO），肿瘤放射研究国家中心（NCRO），海德堡，德国）；National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and the University Medical Center Heidelberg, 69120 Heidelberg, Germany（肿瘤疾病国家中心（NCT），海德堡NCT，德克萨斯儿童医学研究中心（DKFZ）与海德堡大学医学中心之间的合作，海德堡，德国）
MONICA: Benchmarking on Long-tailed Medical Image Classification
Monash University（莫纳什大学）；University College London（伦敦大学学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；Airdoc Research（Airdoc研究院）；Imperial College London（伦敦帝国理工学院）
The University of Hong Kong（香港大学）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）深圳大数据研究院）；Longgang Central Hospital of Shenzhen（深圳龙岗中心医院）；Longgang People Hospital of Shenzhen（深圳龙岗人民医院）
Despite the considerable progress in automatic abdominal multi-organ segmentation from CT/MRI scans in recent years, a comprehensive evaluation of the models' capabilities is hampered by the lack of a large-scale benchmark from diverse clinical scenarios. Constraint by the high cost of collecting and labeling 3D medical data, most of the deep learning models to date are driven by datasets with a limited number of organs of interest or samples, which still limits the power of modern deep models and makes it difficult to provide a fully comprehensive and fair estimate of various methods. To miti…
Transformer, a model comprising attention-based encoder-decoder architecture, have gained prevalence in the field of natural language processing (NLP) and recently influenced the computer vision (CV) space. The similarities between computer vision and medical imaging, reviewed the question among researchers if the impact of transformers on computer vision be translated to medical imaging? In this paper, we attempt to provide a comprehensive and recent review on the application of transformers in medical imaging by; describing the transformer model comparing it with a diversity of convolutional…
Comprehensive Review of Reinforcement Learning for Medical Ultrasound Imaging
Concordia University（康考迪亚大学）；Ecole de Technologie Superieure (ETS)（高等技术学院）；Laval University（拉瓦尔大学）；Khalifa University（哈利法大学）；Lebanese American University（黎巴嫩美国大学）
A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；University of Rouen Normandy（鲁昂诺曼底大学）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）
Medical Image Registration and Its Application in Retinal Images: A Review
Southern University of Science and Technology（南方科技大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Singapore Eye Research Institute（新加坡眼科研究所）
Prompt Mechanisms in Medical Imaging: A Comprehensive Survey
Faculty of Applied Sciences, Macao Polytechnic University（应用科学学院，澳门理工学院）；Netherlands Cancer Institute, Department of Radiology（荷兰癌症研究所，放射科）；Radboud University Medical Centre, Department of Radiology and Nuclear Medicine（拉德堡德大学医学中心，放射科和核医学科）；College of Aerospace Science and Engineering, National University of Defense Technology（航空航天科学与工程学院，国防科技大学）；Medical Department of Breast Cancer, Hunan Cancer Hospital（乳腺癌医学部，湖南癌症医院）；the Affiliated Cancer Hospital of Xiangya School of Medicine, Central South University（湘雅医学院附属肿瘤医院，中南大学）；Faculty of Biomedical Engineering, Eindhoven University of Technology（生物医学工程学院，埃因霍温理工大学）；Laboratory of Advanced Theranostic Materials and Technology, University of Chinese Academy of Sciences（先进诊疗材料与技术实验室，中国科学院大学）
The University of Texas Arlington（德克萨斯大学阿灵顿分校）；RIKEN（理化学研究所）；University of Tokyo（东京大学）；National Cancer Center Research Institute（国立癌症中心研究所）；National Library of Medicine - National Institutes of Health（美国国立卫生研究院国家医学图书馆）；Temple University（天普大学）
Medical visual question answering (VQA) aims to answer clinically relevant questions regarding input medical images. This technique has the potential to improve the efficiency of medical professionals while relieving the burden on the public health system, particularly in resource-poor countries. Existing medical VQA methods tend to encode medical images and learn the correspondence between visual features and questions without exploiting the spatial, semantic, or medical knowledge behind them. This is partially because of the small size of the current medical VQA dataset, which often includes…
UniH$^3$: Unifying Hierarchical Homogeneity and Heterogeneity for All-in-One Medical Image Restoration
School of Biological Science and Medical Engineering, Beihang University（北京航空航天大学生物科学与医学工程学院）；Department of Biomedical Engineering, Tsinghua University（清华大学生物医学工程系）
A Comprehensive Review of Techniques, Algorithms, Advancements, Challenges, and Clinical Applications of Multi-modal Medical Image Fusion for Improved Diagnosis
Interdisciplinary Research Center for Finance and Digital Economy, King Fahd University of Petroleum and Minerals（金融与数字经济交叉研究中心，国王法赫德石油和矿物大学）；Department of Software Engineering, Faculty of Information Technology, Al-Ahliyya Amman University（软件工程系，信息科技学院，阿尔阿赫利亚大学）；Department of Information Systems and Operations Management, King Fahd University of Petroleum and Minerals（信息系统与运营管理系，国王法赫德石油和矿物大学）；ADAPT Research Centre, School of Computer Science, University of Galway（ADAPT研究中心，计算机科学学院，Galway大学）；Department of Mechanical and Nuclear Engineering, Khalifa University（机械与核工程系，哈利法大学）
Palo Alto High School（帕洛阿尔托高中）；Lunit Inc.（Lunit 公司）；KAIST（韩国科学技术院）；Adobe Research（Adobe 研究院）；Stanford University（斯坦福大学）
When the trained physician interprets medical images, they understand the clinical importance of visual features. By applying cognitive attention, they apply greater focus onto clinically relevant regions while disregarding unnecessary features. The use of computer vision to automate the classification of medical images is widely studied. However, the standard convolutional neural network (CNN) does not necessarily employ subconscious feature relevancy evaluation techniques similar to the trained medical specialist and evaluates features more generally. Self-attention mechanisms enable CNNs to…
Imperial College London（帝国理工学院）；Alan Turing Institute（艾伦·图灵研究所）；TECNALIA（泰克纳利亚研究中心）；Basque Research & Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA））
Data quality is the key factor for the development of trustworthy AI in healthcare. A large volume of curated datasets with controlled confounding factors can help improve the accuracy, robustness and privacy of downstream AI algorithms. However, access to good quality datasets is limited by the technical difficulty of data acquisition and large-scale sharing of healthcare data is hindered by strict ethical restrictions. Data synthesis algorithms, which generate data with a similar distribution as real clinical data, can serve as a potential solution to address the scarcity of good quality dat…
West China Hospital, Sichuan University（四川大学华西医院）；Sichuan University（四川大学）；BUPT（北京邮电大学）；Shanghai AI-Lab（上海人工智能实验室）
The large-scale pre-trained vision language models (VLM) have shown remarkable domain transfer capability on natural images. However, it remains unknown whether this capability can also apply to the medical image domain. This paper thoroughly studies the knowledge transferability of pre-trained VLMs to the medical domain, where we show that well-designed medical prompts are the key to elicit knowledge from pre-trained VLMs. We demonstrate that by prompting with expressive attributes that are shared between domains, the VLM can carry the knowledge across domains and improve its generalization. …
Shiraz University of Technology（设拉子理工大学）；Shiraz University of Medical Science（设拉子医科大学）
Image-guided interventions are saving the lives of a large number of patients where the image registration problem should indeed be considered as the most complex and complicated issue to be tackled. On the other hand, the recently huge progress in the field of machine learning made by the possibility of implementing deep neural networks on the contemporary many-core GPUs opened up a promising window to challenge with many medical applications, where the registration is not an exception. In this paper, a comprehensive review on the state-of-the-art literature known as medical image registratio…
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Nanjing University（南京大学）；East China Normal University（华东师范大学）；Fudan University（复旦大学）；Xiamen University（厦门大学）；Monash University（莫纳什大学）；University of Washington（华盛顿大学）；University of Cambridge（剑桥大学）；Stanford University（斯坦福大学）
针对通用 AI 缺乏医学知识的问题，构建 GMAI-VL-5.5M 多模态医学数据集并训练 GMAI-VL 模型，通过三阶段训练整合视觉文本信息，在医学视觉问答和图像诊断等任务上达到最先进性能。
