Computational Anatomy for Multi-Organ Analysis in Medical Imaging: A Review
Imperial College London（帝国理工学院）；Universitat Pompeu Fabra（庞培法布拉大学）；Galgo Medical S.L.（Galgo医疗有限公司）；Nara Institute of Science and Technology (NAIST)（奈良先端科学技术大学院大学（NAIST））；Institució Catalana de Recerca i Estudis Avancats (ICREA)（加泰罗尼亚研究与高级研究机构（ICREA））；Children’s National Health System（儿童国家卫生系统）；George Washington University（乔治华盛顿大学）
Juan J. Cerrolaza, Mirella Lopez-Picazo, Ludovic Humbert, Yoshinobu Sato, Daniel Rueckert, Miguel Angel Gonzalez Ballester, Marius George Linguraru
The medical image analysis field has traditionally been focused on the development of organ-, and disease-specific methods. Recently, the interest in the development of more 20 comprehensive computational anatomical models has grown, leading to the creation of multi-organ models. Multi-organ approaches, unlike traditional organ-specific strategies, incorporate inter-organ relations into the model, thus leading to a more accurate representation of the complex human anatomy. Inter-organ relations are not only spatial, but also functional and physiological. Over the years, the strategies 25 proposed to efficiently model multi-organ structures have evolved from the simple global modeling, to more sophisticated approaches such as sequential, hierarchical, or machine learning-based models. In this paper, we present a review of the state of the art on multi-organ analysis and associated computation anatomy methodology. The manuscript follows a methodology-based classification of the different techniques 30 available for the analysis of multi-organs and multi-anatomical structures, from techniques using point distribution models to the most recent deep learning-based approaches. With more than 300 papers included in this review, we reflect on the trends and challenges of the field of computational anatomy, the particularities of each anatomical region, and the potential of multi-organ analysis to increase the impact of 35 medical imaging applications on the future of healthcare.
Automated classification of human anatomy is an important prerequisite for many computer-aided diagnosis systems. The spatial complexity and variability of anatomy throughout the human body makes classification difficult. "Deep learning" methods such as convolutional networks (ConvNets) outperform other state-of-the-art methods in image classification tasks. In this work, we present a method for organ- or body-part-specific anatomical classification of medical images acquired using computed tomography (CT) with ConvNets. We train a ConvNet, using 4,298 separate axial 2D key-images to learn 5 a…
Self-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planes
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Jarvis Research Center, Tencent YouTu Lab（腾讯优图实验室贾维斯研究中心）
SecondOpinion: Anatomy-Aware Gated Reasoning for Efficient Medical Image Analysis
Center for Computational & Data Sciences, Independent University, Bangladesh（孟加拉国独立大学计算与数据科学中心）；Department of Computer Science and Engineering, Independent University, Bangladesh（孟加拉国独立大学计算机科学与工程系）；Graduate School of Engineering, University of Hyogo（兵库大学工程研究生院）
Osaka University Graduate School of Medicine（大阪大学医学研究生院）
Large numbers of labeled medical images are essential for the accurate detection of anomalies, but manual annotation is labor-intensive and time-consuming. Self-supervised learning (SSL) is a training method to learn data-specific features without manual annotation. Several SSL-based models have been employed in medical image anomaly detection. These SSL methods effectively learn representations in several field-specific images, such as natural and industrial product images. However, owing to the requirement of medical expertise, typical SSL-based models are inefficient in medical image anomal…
Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Lehigh University（里海大学）
针对医学图像数据稀缺及现有方法难以准确呈现解剖与病理细节的问题，提出结合解剖-病理提示和视觉码本细粒度图文对齐的合成模型，并在胸部 X 射线数据上验证其语义准确性。
ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, USTC Suzhou Institute for Advanced Research（医学影像、机器人、分析计算与学习（MIRACLE）实验室，YRD-RIGHT，中国科学技术大学苏州研究院）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology（江苏省多模态数字孪生技术重点实验室）；Biomedical Basic Research Center (BBRC) of Jiangsu Province（江苏省生物医学基础研究中心）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；Anhui IFLYTEK CO., Ltd（安徽科大讯飞股份有限公司）；School of Medicine, Stanford University（医学院，斯坦福大学）；State Key Laboratory of Precision and Intelligent Chemistry, Hefei, Anhui, China（安徽省精密与智能化学重点实验室，合肥，安徽，中国）
AbdomenGen: Sequential Volume-Conditioned Diffusion Framework for Abdominal Anatomy Generation
Center for Virtual Imaging Trials, Dept. of Radiology, Duke University School of Medicine（虚拟成像试验中心，放射科，杜克大学医学部）；Dept. of Electrical and Computer Engineering, Pratt School of Engineering, Duke University（电气与计算机工程系，普拉特工程学院，杜克大学）；Dept. of Physics, Trinity School of Arts and Sciences, Duke University（物理系，特里尼蒂艺术与科学学院，杜克大学）
AbdomenGen提出了一种序列体积条件扩散框架，通过Volume Control Scalar实现可控的腹部解剖生成，实现了高几何保真度和多器官解耦调节。
University of Waterloo（滑铁卢大学）；Suez Canal University（苏伊士运河大学）
Quantifying the accuracy of segmentation and manual delineation of organs, tissue types and tumors in medical images is a necessary measurement that suffers from multiple problems. One major shortcoming of all accuracy measures is that they neglect the anatomical significance or relevance of different zones within a given segment. Hence, existing accuracy metrics measure the overlap of a given segment with a ground-truth without any anatomical discrimination inside the segment. For instance, if we understand the rectal wall or urethral sphincter as anatomical zones, then current accuracy measu…
Anatomy-aware Fine-grained Multimodal Fusion for Laryngopharyngeal Cancer T-Staging Prediction Using CT and Radiology Report
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院北京协和医院）；Hupan Laboratory（湖畔实验室）；Fuzhou University（福州大学）；Eye & ENT Hospital, Fudan University（复旦大学附属眼耳鼻喉科医院）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；Chang Gung Memorial Hospital（长庚纪念医院）；Zhongshan Hospital(Xiamen), Fudan University（复旦大学附属中山医院厦门医院）
University of Oxford（牛津大学）；Tel-Aviv University（特拉维夫大学）
Self-supervised contrastive representation learning offers the advantage of learning meaningful visual representations from unlabeled medical datasets for transfer learning. However, applying current contrastive learning approaches to medical data without considering its domain-specific anatomical characteristics may lead to visual representations that are inconsistent in appearance and semantics. In this paper, we propose to improve visual representations of medical images via anatomy-aware contrastive learning (AWCL), which incorporates anatomy information to augment the positive/negative pa…
CT-GRAPH: Hierarchical Graph Attention Network for Anatomy-Guided CT Report Generation
Institute for AI in Medicine (IKIM), University Hospital Essen (AöR)（人工智能医学研究所（IKIM）、埃森大学医院（AöR））；Department of Nuclear Medicine, University Hospital Essen (AöR)（核医学系、埃森大学医院（AöR））；Department of Physics, TU Dortmund（物理系、多特蒙德技术大学）
University of Pittsburgh（匹兹堡大学）；Boston University（波士顿大学）
Large-scale volumetric medical images with annotation are rare, costly, and time prohibitive to acquire. Self-supervised learning (SSL) offers a promising pre-training and feature extraction solution for many downstream tasks, as it only uses unlabeled data. Recently, SSL methods based on instance discrimination have gained popularity in the medical imaging domain. However, SSL pre-trained encoders may use many clues in the image to discriminate an instance that are not necessarily disease-related. Moreover, pathological patterns are often subtle and heterogeneous, requiring the ability of the…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Institute for AI in Medicine, University Hospital Essen（埃森大学医院人工智能医学研究所）；University of Duisburg-Essen（杜伊斯堡-埃森大学）；German Cancer Consortium (DKTK)（德国癌症联盟）
Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Georgia Institute of Technology（佐治亚理工学院）
Thoracic disease detection from chest radiographs using deep learning methods has been an active area of research in the last decade. Most previous methods attempt to focus on the diseased organs of the image by identifying spatial regions responsible for significant contributions to the model's prediction. In contrast, expert radiologists first locate the prominent anatomical structures before determining if those regions are anomalous. Therefore, integrating anatomical knowledge within deep learning models could bring substantial improvement in automatic disease classification. Motivated by …
See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancement
Johns Hopkins University（约翰霍普金斯大学）；University of Copenhagen（哥本哈根大学）；University of Virginia（弗吉尼亚大学）；University of Bologna（博洛尼亚大学）；Italian Institute of Technology（意大利理工学院）；University of Zurich（苏黎世大学）；ETH AI Center（ETH人工智能中心）；Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；Jagiellonian University（雅盖隆大学）；NVIDIA（NVIDIA公司）；Johns Hopkins Medicine（约翰霍普金斯医学）；University of California, San Francisco（旧金山大学）
seg2med: a bridge from artificial anatomy to multimodal medical images
Computer Assisted Clinical Medicine, Medical Faculty Mannheim, Heidelberg University（计算机辅助临床医学，曼海姆医学院，海德堡大学）；Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg（模式识别实验室，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Department of Radiology and Nuclear Medicine, University Medical Center Mannheim（放射学与核医学系，曼海姆大学医学中心）；Mannheim Institute for Intelligent Systems in Medicine, Medical Faculty Mannheim, Heidelberg University（曼海姆智能医学研究所，曼海姆医学院，海德堡大学）；Optical Bioimaging Laboratory, Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore（光学生物成像实验室，生物医学工程系，设计与工程学院，新加坡国立大学）
ORACLE-CT: Anatomy-Aware Support Pooling for CT Classification
Center for Virtual Imaging Trials, RAI Labs, Department of Radiology, Duke University（虚拟成像试验中心，RAI实验室，放射学系，杜克大学）；Electrical and Computer Engineering, Pratt School of Engineering, Duke University（电气与计算机工程，工程学院，杜克大学）；Department of Mathematics, Trinity College of Arts & Sciences, Duke University（数学系，艺术与科学学院，杜克大学）；Department of Radiology and Imaging Sciences, University of Arizona College of Medicine（放射学与影像科学系，亚利桑那大学医学院）
University of Edinburgh（爱丁堡大学）；Edinburgh Imaging Facility QMRI（爱丁堡影像设施QMRI）；Centre for Cardiovascular Science（心血管科学中心）；Cedars Sinai Medical Center（西达赛奈医疗中心）；The Alan Turing Institute（艾伦·图灵研究所）
Typically, a medical image offers spatial information on the anatomy (and pathology) modulated by imaging specific characteristics. Many imaging modalities including Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) can be interpreted in this way. We can venture further and consider that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. Here, we explicitly learn this decomposed (disentangled) representation of imaging data, focusing in particular on cardiac images. We propose Spatial Decomposition Netw…
Utah State University（犹他州立大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Idaho（爱达荷大学）；The Affiliated Hospital of Qingdao University（青岛大学附属医院）；The Second Hospital of Hebei Medical University（河北医科大学第二医院）
Tumor saliency estimation aims to localize tumors by modeling the visual stimuli in medical images. However, it is a challenging task for breast ultrasound due to the complicated anatomic structure of the breast and poor image quality; and existing saliency estimation approaches only model generic visual stimuli, e.g., local and global contrast, location, and feature correlation, and achieve poor performance for tumor saliency estimation. In this paper, we propose a novel optimization model to estimate tumor saliency by utilizing breast anatomy. First, we model breast anatomy and decompose bre…
AG-CRC: Anatomy-Guided Colorectal Cancer Segmentation in CT with Imperfect Anatomical Knowledge
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Digital Medicine Innovation Center（上海数字医学创新中心）；Ruijin Hospital（瑞金医院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）
Vision transformers, with their ability to more efficiently model long-range context, have demonstrated impressive accuracy gains in several computer vision and medical image analysis tasks including segmentation. However, such methods need large labeled datasets for training, which is hard to obtain for medical image analysis. Self-supervised learning (SSL) has demonstrated success in medical image segmentation using convolutional networks. In this work, we developed a \underline{s}elf-distillation learning with \underline{m}asked \underline{i}mage modeling method to perform SSL for vision \u…
Convolutional neural networks demonstrated outstanding empirical results in computer vision and speech recognition tasks where labeled training data is abundant. In medical imaging, there is a huge variety of possible imaging modalities and contrasts, where annotated data is usually very scarce. We present two approaches to deal with this challenge. A network pretrained in a different domain with abundant data is used as a feature extractor, while a subsequent classifier is trained on a small target dataset; and a deep architecture trained with heavy augmentation and equipped with sophisticate…
Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The First Affiliated Hospital of College of Medicine, Zhejiang University（浙江大学医学院附属第一医院）；Hupan Lab（虎盘实验室）
Aligning Fetal Anatomy with Kinematic Tree Log-Euclidean PolyRigid Transforms
Computer Science and Artificial Intelligence Lab, MIT, Cambridge, USA（麻省理工学院计算机科学与人工智能实验室）；California Institute of Technology, Pasadena, USA（加州理工学院）；Boston Children’s Hospital and Harvard Medical School, Boston, USA（哈佛医学院波士顿儿童医院）；The Hebrew University of Jerusalem, Jerusalem, Israel（耶路撒冷希伯来大学）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Daimler TSS（戴姆勒TSS公司）；University Hospital Heidelberg（海德堡大学医院）；German Cancer Research Center（德国癌症研究中心）；University Clinic Essen（埃森大学医院）
In clinical radiology reports, doctors capture important information about the patient's health status. They convey their observations from raw medical imaging data about the inner structures of a patient. As such, formulating reports requires medical experts to possess wide-ranging knowledge about anatomical regions with their normal, healthy appearance as well as the ability to recognize abnormalities. This explicit grasp on both the patient's anatomy and their appearance is missing in current medical image-processing systems as annotations are especially difficult to gather. This renders th…
CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University（斯坦福大学医学与影像人工智能中心）；Department of Radiology, Stanford University（斯坦福大学放射学系）
Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding
DAMO Academy, Alibaba Group（达摩院（阿里巴巴集团））；The First Affiliated Hospital of College of Medicine, Zhejiang University（浙江大学医学院附属第一医院）；Zhejiang University（浙江大学）；Westlake University（西湖大学）
ETH Zürich（苏黎世联邦理工学院）；University of Cambridge（剑桥大学）
Semantic segmentation of medical images is a crucial step for the quantification of healthy anatomy and diseases alike. The majority of the current state-of-the-art segmentation algorithms are based on deep neural networks and rely on large datasets with full pixel-wise annotations. Producing such annotations can often only be done by medical professionals and requires large amounts of valuable time. Training a medical image segmentation network with weak annotations remains a relatively unexplored topic. In this work we investigate training strategies to learn the parameters of a pixel-wise s…
Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Tsinghua University（清华大学）
Automated radiology report generation aims at automatically generating a detailed description of medical images, which can greatly alleviate the workload of radiologists and provide better medical services to remote areas. Most existing works pay attention to the holistic impression of medical images, failing to utilize important anatomy information. However, in actual clinical practice, radiologists usually locate important anatomical structures, and then look for signs of abnormalities in certain structures and reason the underlying disease. In this paper, we propose a novel framework AGFNet…
Johns Hopkins University（约翰斯·霍普金斯大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；The Johns Hopkins Hospital（约翰斯·霍普金斯医院）
Several methods to perform extrapolation of unknown anatomy were evaluated. The primary application is to enhance surgical procedures that may use partial medical images or medical images of incomplete anatomy. Le Fort-based, face-jaw-teeth transplant is one such procedure. From CT data of 36 skulls and 21 mandibles separate Statistical Shape Models of the anatomical surfaces were created. Using the Statistical Shape Models, incomplete surfaces were projected to obtain complete surface estimates. The surface estimates exhibit non-zero error in regions where the true surface is known; it is des…
MambaControl: Anatomy Graph-Enhanced Mamba ControlNet with Fourier Refinement for Diffusion-Based Disease Trajectory Prediction
Faculty of Applied Sciences, Macao Polytechnic University（应用科学学院，澳门理工学院）；Department of Electrical Engineering, Zhejiang University（电气工程学院，浙江大学）；Department of Computer Science, The University of Adelaide（计算机科学学院，阿德莱德大学）；Department of Computer Science, University of Birmingham（计算机科学学院，伯明翰大学）
University of Edinburgh（爱丁堡大学）；Canon Medical Research Europe Ltd.（佳能医疗欧洲研究有限公司）；AC Codewheel Ltd（AC Codewheel有限公司）；The Alan Turing Institute（艾伦·图灵研究所）
Training medical image segmentation models usually requires a large amount of labeled data. By contrast, humans can quickly learn to accurately recognise anatomy of interest from medical (e.g. MRI and CT) images with some limited guidance. Such recognition ability can easily generalise to new images from different clinical centres. This rapid and generalisable learning ability is mostly due to the compositional structure of image patterns in the human brain, which is less incorporated in medical image segmentation. In this paper, we model the compositional components (i.e. patterns) of human a…
End-to-End Learning of Multi-Organ Implicit Surfaces from 3D Medical Imaging Data
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, France Department of Diagnostic Radiology, McGill University, Montreal, Canada Augmented Intelligence \& Precision Health Laboratory (AIPHL), McGill University Health Centre Research Institute, Montreal, Canada
Medical Resonance Imaging or MRI is a medical image processing technique that used radio waves to scan the body. It is a tomographic imaging technique, principally used in the field of radiology. With the advantage of being a painless diagnostic procedure, MRI allows medical personnel to illustrate clear pictures of the anatomy and the physiological processes occurring in the body, thus allowing early detection and treatment of diseases. These images, combined with image processing techniques may be used in the detection of tumors, difficult to identify with the naked eye. This digital assignm…
Medical Image Analysis is currently experiencing a paradigm shift due to Deep Learning. This technology has recently attracted so much interest of the Medical Imaging community that it led to a specialized conference in `Medical Imaging with Deep Learning' in the year 2018. This article surveys the recent developments in this direction, and provides a critical review of the related major aspects. We organize the reviewed literature according to the underlying Pattern Recognition tasks, and further sub-categorize it following a taxonomy based on human anatomy. This article does not assume prior…
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Medical images are naturally associated with rich semantics about the human anatomy, reflected in an abundance of recurring anatomical patterns, offering unique potential to foster deep semantic representation learning and yield semantically more powerful models for different medical applications. But how exactly such strong yet free semantics embedded in medical images can be harnessed for self-supervised learning remains largely unexplored. To this end, we train deep models to learn semantically enriched visual representation by self-discovery, self-classification, and self-restoration of th…
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
Creating a large-scale dataset of abnormality annotation on medical images is a labor-intensive and costly task. Leveraging weak supervision from readily available data such as radiology reports can compensate lack of large-scale data for anomaly detection methods. However, most of the current methods only use image-level pathological observations, failing to utilize the relevant anatomy mentions in reports. Furthermore, Natural Language Processing (NLP)-mined weak labels are noisy due to label sparsity and linguistic ambiguity. We propose an Anatomy-Guided chest X-ray Network (AGXNet) to addr…
