Transfer learning from natural images to medical images has been established as one of the most practical paradigms in deep learning for medical image analysis. To fit this paradigm, however, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information, thereby inevitably compromising its performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learnt by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch and existing pre-trained 3D models in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D/2.5D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated and recurrent anatomy in medical images can serve as strong yet free supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all codes and pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Transfer learning from natural image to medical image has established as one of the most practical paradigms in deep learning for medical image analysis. However, to fit this paradigm, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information and inevitably compromising the performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learned by self-…
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Medical images are naturally associated with rich semantics about the human anatomy, reflected in an abundance of recurring anatomical patterns, offering unique potential to foster deep semantic representation learning and yield semantically more powerful models for different medical applications. But how exactly such strong yet free semantics embedded in medical images can be harnessed for self-supervised learning remains largely unexplored. To this end, we train deep models to learn semantically enriched visual representation by self-discovery, self-classification, and self-restoration of th…
Relational Anatomical Supervision for Accurate 3D Multi-Chamber Cardiac Mesh Reconstruction
Bioengineering Department Imperial College London, London W12 7SL, United Kingdom；Lung Institute, Imperial College London, London, United Kingdom；Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, United Kingdom；School of Biomedical Engineering \& Imaging Sciences, King's College London, London WC2R 2LS, United Kingdom；ROAS Thrust, Hong Kong University of Science；Department of Electronic；Computer Engineering, Hong Kong University of Science
Mask-HybridGNet: Graph-based segmentation with emergent anatomical correspondence from pixel-level supervision
Biomedical Image Technologies, ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain（生物医学影像技术，电信工程学院，马德里理工大学，马德里，西班牙）；IHU PRISM National Precision Medicine Center in Oncology, Cancer Data Science Unit, CentraleSupélec, Gustave Roussy, INSERM, Université Paris-Saclay, Gif-sur-Yvette, France（IHU PRISM国家精准医学中心（肿瘤学），癌症数据科学单元，中央圣艾尔布兰，格斯塔夫·罗西，国家医学研究所，巴黎-萨克雷大学，法国）
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
This paper introduces a new concept called "transferable visual words" (TransVW), aiming to achieve annotation efficiency for deep learning in medical image analysis. Medical imaging--focusing on particular parts of the body for defined clinical purposes--generates images of great similarity in anatomy across patients and yields sophisticated anatomical patterns across images, which are associated with rich semantics about human anatomy and which are natural visual words. We show that these visual words can be automatically harvested according to anatomical consistency via self-discovery, and …
CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University（斯坦福大学医学与影像人工智能中心）；Department of Radiology, Stanford University（斯坦福大学放射学系）
Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
PAII Inc.（PAII公司）；The First Affiliated Hospital Zhejiang University（浙江大学附属第一医院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Ping An Insurance Company of China（中国平安保险（集团）股份有限公司）；Zhejiang Provincial People’s Hospital（浙江省人民医院）
Lymph node station (LNS) delineation from computed tomography (CT) scans is an indispensable step in radiation oncology workflow. High inter-user variabilities across oncologists and prohibitive laboring costs motivated the automated approach. Previous works exploit anatomical priors to infer LNS based on predefined ad-hoc margins. However, without voxel-level supervision, the performance is severely limited. LNS is highly context-dependent - LNS boundaries are constrained by anatomical organs - we formulate it as a deep spatial and contextual parsing problem via encoded anatomical organs. Thi…
University of Electronic Science and Technology of China（电子科技大学）
Localization of anatomical landmarks is essential for clinical diagnosis, treatment planning, and research. In this paper, we propose a novel deep network, named feature aggregation and refinement network (FARNet), for the automatic detection of anatomical landmarks. To alleviate the problem of limited training data in the medical domain, our network adopts a deep network pre-trained on natural images as the backbone network and several popular networks have been compared. Our FARNet also includes a multi-scale feature aggregation module for multi-scale feature fusion and a feature refinement …
Anomaly detection and localization in medical images is a challenging task, especially when the anomaly exhibits a change of existing structures, e.g., brain atrophy or changes in the pleural space due to pleural effusions. In this work, we present a weakly supervised and detail-preserving method that is able to detect structural changes of existing anatomical structures. In contrast to standard anomaly detection methods, our method extracts information about the disease characteristics from two groups: a group of patients affected by the same disease and a healthy control group. Together with…
Phi-SegNet: Phase-Integrated Supervision for Medical Image Segmentation
mHealth Lab, Department of Biomedical Engineering, Bangladesh University of Engineering and Technology（孟加拉工程与技术大学生物医学工程系mHealth实验室）；Department of Electrical and Computer Engineering, Johns Hopkins University（约翰霍普金斯大学电气与计算机工程系）；Center for Bioengineering Innovation and Design, Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物工程创新与设计中心，生物医学工程系）
University Hospitals Leuven（鲁汶大学医院）；University Hospitals Tanta（坦塔大学医院）；University College London（伦敦大学学院）
Deep neural networks have increased the accuracy of automatic segmentation, however, their accuracy depends on the availability of a large number of fully segmented images. Methods to train deep neural networks using images for which some, but not all, regions of interest are segmented are necessary to make better use of partially annotated datasets. In this paper, we propose the first axiomatic definition of label-set loss functions that are the loss functions that can handle partially segmented images. We prove that there is one and only one method to convert a classical loss function for fu…
AnyCXR: Human Anatomy Segmentation of Chest X-ray at Any Acquisition Position using Multi-stage Domain Randomized Synthetic Data with Imperfect Annotations and Conditional Joint Annotation Regularization Learning
Department of Radiology, Northwestern University, Chicago, IL, USA（西北大学放射科）；Data Science Institute, Vanderbilt University, Nashville, TN, USA（范德比尔特大学数据科学研究院）；Department of Orthopedics, The Second Hospital of Shanxi Medical University, Taiyuan, Shanxi, P.R. China（山西医科大学第二医院骨科）；Second Clinical Medical College, Shanxi Medical University, Taiyuan, Shanxi, P.R. China（山西医科大学第二临床医学院）；School of Computer Science, University of Sydney, Sydney, NSW, Australia（悉尼大学计算机科学学院）
HYATT-Net is Grand: A Hybrid Attention Network for Performant Anatomical Landmark Detection
Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC)（中国科学技术大学苏州高等研究院）；School of Computer Science and Technology, University of Science and Technology of China (USTC)（中国科学技术大学计算机科学与技术学院）；School of Information Science and Technology, Eastern Institute of Technology (EIT)（宁波东方理工大学信息科学与技术学院）；Stanford University（斯坦福大学）
Removing Motion Artifact in MRI by Using a Perceptual Loss Driven Deep Learning Framework
Institute of Magnetic Resonance and Molecular Imaging in Medicine（医学磁共振与分子成像研究所）；Shanghai Key Laboratory of Magnetic Resonance（上海磁共振重点实验室）；The First Affiliated Hospital of Naval Medical University Department of Radiology（海军医学院第一附属医院放射科）；Laboratory for Intelligent Medical Imaging（智能医学成像实验室）；Tsinghua Cross-Strait Research Institute（清华大学海峡研究院）；RadioDynamic Medical（RadioDynamic医疗）
Learning from Acquisition: Metadata-driven Multimodal Pre-training for Cardiac MRI
Department of Surgery & Cancer（外科与癌症系）；Bioengineering Department and Imperial-X（生物工程系和Imperial-X）；National Heart and Lung Institute（国家心脏和肺研究所）；Cardiovascular Research Centre（心血管研究中心）；School of Biomedical Engineering & Imaging Sciences（生物医学工程与成像科学学院）
Semi-supervised learning (semi-SL) is a promising alternative to supervised learning for medical image analysis when obtaining good quality supervision for medical imaging is difficult. However, semi-SL assumes that the underlying distribution of unaudited data matches that of the few labeled samples, which is often violated in practical settings, particularly in medical images. The presence of out-of-distribution (OOD) samples in the unlabeled training pool of semi-SL is inevitable and can reduce the efficiency of the algorithm. Common preprocessing methods to filter out outlier samples may n…
Closed-loop coupling of personalised and foundation models for real-time treatment guidance with MRI
Image X Institute, Sydney School of Health Sciences, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia（悉尼大学医学与健康学院悉尼健康科学学院Image X研究所，悉尼，澳大利亚）；GenesisCare, Sydney, Australia（GenesisCare，悉尼，澳大利亚）；School of Clinical Medicine, Medicine & Health, University of New South Wales, Sydney, Australia（新南威尔士大学医学与健康学院临床医学系，悉尼，澳大利亚）；Centre for Medical Radiation Physics, School of Mathematics and Physics, University of Wollongong, Wollongong, Australia（伍伦贡大学数学与物理学院医学辐射物理中心，伍伦贡，澳大利亚）
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Fudan University, China（复旦大学）；University of Queensland, Australia（昆士兰大学）；Shanghai Academy of AI for Science, China（上海人工智能科学研究院）；Huashan Hospital, National Center for Neurological Disorders, Fudan University, China（华山医院，国家神经系统疾病中心，复旦大学）；Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore（生物信息研究所（BII），科技研究局（A*STAR），新加坡）
MedLSAM: Localize and Segment Anything Model for 3D CT Images
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；Shanghai AI Lab（上海人工智能实验室）；West China Hospital, Sichuan University（四川大学华西医院）
ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, USTC Suzhou Institute for Advanced Research（医学影像、机器人、分析计算与学习（MIRACLE）实验室，YRD-RIGHT，中国科学技术大学苏州研究院）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology（江苏省多模态数字孪生技术重点实验室）；Biomedical Basic Research Center (BBRC) of Jiangsu Province（江苏省生物医学基础研究中心）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；Anhui IFLYTEK CO., Ltd（安徽科大讯飞股份有限公司）；School of Medicine, Stanford University（医学院，斯坦福大学）；State Key Laboratory of Precision and Intelligent Chemistry, Hefei, Anhui, China（安徽省精密与智能化学重点实验室，合肥，安徽，中国）
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
Creating a large-scale dataset of abnormality annotation on medical images is a labor-intensive and costly task. Leveraging weak supervision from readily available data such as radiology reports can compensate lack of large-scale data for anomaly detection methods. However, most of the current methods only use image-level pathological observations, failing to utilize the relevant anatomy mentions in reports. Furthermore, Natural Language Processing (NLP)-mined weak labels are noisy due to label sparsity and linguistic ambiguity. We propose an Anatomy-Guided chest X-ray Network (AGXNet) to addr…
Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction
Dept. of Electrical and Electronics Engineering, and National Magnetic Resonance Research Center (UMRAM), Bilkent University（电气与电子工程系，国家磁共振研究中心（UMRAM），比尔肯大学）
Yale University（耶鲁大学）；New York University（纽约大学）；Hyperfine Research（Hyperfine研究公司）
Multi-contrast MRI (MC-MRI) captures multiple complementary imaging modalities to aid in radiological decision-making. Given the need for lowering the time cost of multiple acquisitions, current deep accelerated MRI reconstruction networks focus on exploiting the redundancy between multiple contrasts. However, existing works are largely supervised with paired data and/or prohibitively expensive fully-sampled MRI sequences. Further, reconstruction networks typically rely on convolutional architectures which are limited in their capacity to model long-range interactions and may lead to suboptima…
Asymmetric Paired-Annotation Learning for Multi-Structure ULF Pediatric Brain MRI Segmentation
VinUniversity；College of Engineering & Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）；Hanoi University of Science and Technology（河内科技大学）；University of Information Technology, VNU-HCM（胡志明市国家大学信息技术大学）；Carnegie Mellon University（卡内基梅隆大学）；Northwestern University（西北大学）；Center for Innovations in Health Sciences, VinUniversity（VinUniversity健康科学创新中心）
针对0.064 T ULF儿科脑部MRI分割难题，提出基于nnU-Net的非对称监督策略AURA，利用两类非等效标注训练，在16例数据上取得优于基线的分割性能。
Beyond the LUMIR challenge: The pathway to foundational registration models
The Russell H. Morgan Department of Radiology（Russell H. Morgan放射科）；Radiological Science, Johns Hopkins Medical School（约翰霍普金斯医学院放射科学）；Department of Computer Science, Aalto University（阿尔托大学计算机科学系）；Cornell University（康奈尔大学）；Canon Medical Systems (China) Co. Ltd.（佳能医疗系统（中国）有限公司）；School of Biomedical Engineering, Shenzhen University Medical School（深圳大学医学院生物医学工程学院）；Department of Imaging Physics, Delft University of Technology（代尔夫特理工大学成像物理系）；Technical University of Munich（慕尼黑技术大学）；Radboud University Medical Center（拉德伯德大学医学中心）；Inria, Paris, France（法国巴黎Inria）
Cyclic Self-Supervised Diffusion for Ultra Low-field to High-field MRI Synthesis
Department of Bioengineering and Imperial-X, Imperial College London（生物工程系和Imperial-X，帝国理工学院伦敦分校）；Department of Bioengineering, Imperial College London（生物工程系，帝国理工学院伦敦分校）；School of Biomedical Engineering, Sun Yat-sen University（生物医学工程学院，中山大学）；Monash Biomedical Imaging, Monash University（莫纳什大学生物医学成像）
REACT-KD: Region-Aware Cross-modal Topological Knowledge Distillation for Interpretable Medical Image Classification
Department of Health Technology and Informatics, Hong Kong Polytechnic University（健康科技与信息技术系，香港理工大学）；Department of Nuclear Medicine and PET, Hong Kong Sanatorium and Hospital（核医学与PET部门，香港疗养院及医院）
Recent advances in deep learning have achieved promising performance for medical image analysis, while in most cases ground-truth annotations from human experts are necessary to train the deep model. In practice, such annotations are expensive to collect and can be scarce for medical imaging applications. Therefore, there is significant interest in learning representations from unlabelled raw data. In this paper, we propose a self-supervised learning approach to learn meaningful and transferable representations from medical imaging video without any type of human annotation. We assume that in …
Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports
DAMO Academy, Alibaba Group（阿里巴巴集团达摩学院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Hupan Lab（湖畔实验室）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；South China Hospital, Medical School, Shenzhen University（深圳大学医学院华南医院）
State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Vision Science, Guangdong Provincial Clinical Research Center for Ocular Diseases（眼科学国家重点实验室、中山眼科中心、中山大学、广东省眼科学重点实验室和视觉科学、广东省眼科临床研究中心）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Hainan International College, Minzu University of China（海南国际学院，中国民族大学）；School of Information Technology, Faculty of Business and Hospitality, Torrens University Australia（澳大利亚托伦斯大学信息科技学院，商业与酒店管理学院）；AIM for Health Lab, Faculty of IT, Monash University, Australia（健康AIM实验室，墨尔本大学IT学院，澳大利亚）；Department of Ophthalmology, Guangzhou First People’s Hospital, the Second Affiliated Hospital of South China University of Technology（广州第一人民医院，华南理工大学第二附属医院）；The Third Xiangya Hospital of Central South University（中南大学湘雅医学院第三医院）
