Sparsity of annotated data is a major limitation in medical image processing tasks such as registration. Registered multimodal image data are essential for the diagnosis of medical conditions and the success of interventional medical procedures. To overcome the shortage of data, we present a method that allows the generation of annotated multimodal 4D datasets. We use a CycleGAN network architecture to generate multimodal synthetic data from the 4D extended cardiac-torso (XCAT) phantom and real patient data. Organ masks are provided by the XCAT phantom, therefore the generated dataset can serve as ground truth for image segmentation and registration. Realistic simulation of respiration and heartbeat is possible within the XCAT framework. To underline the usability as a registration ground truth, a proof of principle registration is performed. Compared to real patient data, the synthetic data showed good agreement regarding the image voxel intensity distribution and the noise characteristics. The generated T1-weighted magnetic resonance imaging (MRI), computed tomography (CT), and cone beam CT (CBCT) images are inherently co-registered. Thus, the synthetic dataset allowed us to optimize registration parameters of a multimodal non-rigid registration, utilizing liver organ masks for evaluation. Our proposed framework provides not only annotated but also multimodal synthetic data which can serve as a ground truth for various tasks in medical imaging processing. We demonstrated the applicability of synthetic data for the development of multimodal medical image registration algorithms.
Human body is a complex dynamic system composed of various sub-dynamic parts. Especially, thoracic and abdominal organs have complex internal shape variations with different frequencies by various reasons such as respiration with fast motion and peristalsis with slower motion. CT protocols for abdominal lesions are multi-phase scans for various tumor detection to use different vascular contrast, however, they are not aligned well enough to visually check the same area. In this paper, we propose a time-efficient and accurate deformable registration algorithm for multi-phase CT scans considering…
Deformable registration of two-dimensional/three-dimensional (2D/3D) images of abdominal organs is a complicated task because the abdominal organs deform significantly and their contours are not detected in two-dimensional X-ray images. We propose a supervised deep learning framework that achieves 2D/3D deformable image registration between 3D volumes and single-viewpoint 2D projected images. The proposed method learns the translation from the target 2D projection images and the initial 3D volume to 3D displacement fields. In experiments, we registered 3D-computed tomography (CT) volumes to di…
SINTEF（挪威科学与工业研究基金会）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）；St. Olavs hospital, Trondheim University Hospital（圣奥拉夫医院，特隆赫姆大学医院）；Oslo University Hospital（奥斯陆大学医院）
Purpose: This study aims to explore training strategies to improve convolutional neural network-based image-to-image deformable registration for abdominal imaging. Methods: Different training strategies, loss functions, and transfer learning schemes were considered. Furthermore, an augmentation layer which generates artificial training image pairs on-the-fly was proposed, in addition to a loss layer that enables dynamic loss weighting. Results: Guiding registration using segmentations in the training step proved beneficial for deep-learning-based image registration. Finetuning the pretrained m…
The Chinese University of Hong Kong（香港中文大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Tencent Jarvis Lab（腾讯医疗健康（深圳）有限公司（腾讯医疗AI实验室/腾讯天衍实验室））；Tsinghua University（清华大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
In order to tackle the difficulty associated with the ill-posed nature of the image registration problem, regularization is often used to constrain the solution space. For most learning-based registration approaches, the regularization usually has a fixed weight and only constrains the spatial transformation. Such convention has two limitations: (i) Besides the laborious grid search for the optimal fixed weight, the regularization strength of a specific image pair should be associated with the content of the images, thus the "one value fits all" training scheme is not ideal; (ii) Only spatiall…
Purpose: Diffeomorphic image registration is essential in many medical imaging applications. Several registration algorithms of such type have been proposed, but primarily for intra-contrast alignment. Currently, efficient inter-modal/contrast diffeomorphic registration, which is vital in numerous applications, remains a challenging task. Methods: We proposed a novel inter-modal/contrast registration algorithm that leverages Robust PaTch-based cOrrelation Ratio (RaPTOR) metric to allow inter-modal/contrast image alignment and bandlimited geodesic shooting demonstrated in Fourier Approximated L…
TotalRegistrator: Towards a Lightweight Foundation Model for CT Image Registration
Department of Imaging, Radboudumc, Nijmegen, the Netherlands（影像部，Radboudumc，尼姆egen，荷兰）；Institute for Medical Informatics, University of Lübeck, Lübeck, Germany（医学信息研究所，吕贝克大学，吕贝克，德国）；Department of Nuclear Medicine, Hospital 108, Hanoi, Vietnam（核医学部，108医院，河内，越南）；Department of Diagnostic Imaging and Interventional Radiology, Thai Nguyen National Hospital, Thai Nguyen, Vietnam（诊断影像和介入放射科，Thai Nguyen国家医院，Thai Nguyen，越南）；Interventional Radiology Center, Tam Anh Hospital, Hanoi, Vietnam（介入放射科中心，Tam Anh医院，河内，越南）；FET, Vietnam National University, University of Engineering and Technology, Hanoi, Vietnam（FET，越南国家大学，工程与技术大学，河内，越南）
Images from Positron Emission Tomography (PET) deliver functional data such as perfusion and metabolism. On the other hand, images from Magnetic Resonance Imaging (MRI) provide information describing anatomical structures. Fusing the complementary information from the two modalities is helpful in oncology. In this project, we implemented a complete tool allowing semi-automatic MRI-PET registration for small animal imaging in the preclinical studies. A two-stage hierarchical registration approach is proposed. First, a global affine registration is applied. For robust and fast registration, prin…
School of Biomedical Engineering and Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与成像科学学院）；Department of Women and Children’s Health, School of Life Course Sciences, King’s College London（伦敦国王学院生命历程科学学院妇幼健康系）
Accurately estimating and correcting the motion artifacts are crucial for 3D image reconstruction of the abdominal and in-utero magnetic resonance imaging (MRI). The state-of-art methods are based on slice-to-volume registration (SVR) where multiple 2D image stacks are acquired in three orthogonal orientations. In this work, we present a novel reconstruction pipeline that only needs one orientation of 2D MRI scans and can reconstruct the full high-resolution image without masking or registration steps. The framework consists of two main stages: the respiratory motion estimation using a self-su…
Image registration aims to establish spatial correspondence across pairs, or groups of images, and is a cornerstone of medical image computing and computer-assisted-interventions. Currently, most deep learning-based registration methods assume that the desired deformation fields are globally smooth and continuous, which is not always valid for real-world scenarios, especially in medical image registration (e.g. cardiac imaging and abdominal imaging). Such a global constraint can lead to artefacts and increased errors at discontinuous tissue interfaces. To tackle this issue, we propose a weakly…
Otto von Guericke University (OVGU)（奥托·冯·格里克大学（OVGU））
Dynamic MRI is a technique of acquiring a series of images continuously to follow the physiological changes over time. However, such fast imaging results in low resolution images. In this work, abdominal deformation model computed from dynamic low resolution images have been applied to high resolution image, acquired previously, to generate dynamic high resolution MRI. Dynamic low resolution images were simulated into different breathing phases (inhale and exhale). Then, the image registration between breathing time points was performed using the B-spline SyN deformable model and using cross-c…
Institute of Medical Informatics, Universität zu Lübeck（吕贝克大学医学信息学研究所）
Though, deep learning based medical image registration is currently starting to show promising advances, often, it still fells behind conventional frameworks in terms of registration accuracy. This is especially true for applications where large deformations exist, such as registration of interpatient abdominal MRI or inhale-to-exhale CT lung registration. Most current works use U-Net-like architectures to predict dense displacement fields from the input images in different supervised and unsupervised settings. We believe that the U-Net architecture itself to some level limits the ability to p…
Spatially Covariant Image Registration with Text Prompts
College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；Cornell University（康奈尔大学）；University of Pennsylvania（宾夕法尼亚大学）；New York University（纽约大学）；Shanghai Jiao Tong University（上海交通大学）
Tsinghua University（清华大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Deformable image registration (DIR) is essential for many image-guided therapies. Recently, deep learning approaches have gained substantial popularity and success in DIR. Most deep learning approaches use the so-called mono-stream "high-to-low, low-to-high" network structure, and can achieve satisfactory overall registration results. However, accurate alignments for some severely deformed local regions, which are crucial for pinpointing surgical targets, are often overlooked. Consequently, these approaches are not sensitive to some hard-to-align regions, e.g., intra-patient registration of de…
CT Synthesis with Conditional Diffusion Models for Abdominal Lymph Node Segmentation
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；Department of Surgical Oncology and General Surgery, Key Laboratory of Precision Diagnosis and Treatment of Gastrointestinal Tumors, Ministry of Education, The First Hospital of China Medical University, Shenyang, China（外科肿瘤科和普通外科，精准诊断与治疗胃肠肿瘤国家重点实验室，教育部，中国医科大学第一附属医院，沈阳，中国）；Department of Radiology, The First Hospital of China Medical University, Shenyang, China（放射科，中国医科大学第一附属医院，沈阳，中国）；Shanghai AI Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）
Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；Graduate School of Medicine, Kyoto University（京都大学医学研究科）
Organ shape reconstruction based on a single-projection image during treatment has wide clinical scope, e.g., in image-guided radiotherapy and surgical guidance. We propose an image-to-graph convolutional network that achieves deformable registration of a 3D organ mesh for a single-viewpoint 2D projection image. This framework enables simultaneous training of two types of transformation: from the 2D projection image to a displacement map, and from the sampled per-vertex feature to a 3D displacement that satisfies the geometrical constraint of the mesh structure. Assuming application to radiati…
University of Sydney（悉尼大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
White matter (WM) tract segmentation based on diffusion magnetic resonance imaging (dMRI) plays an important role in the analysis of human health and brain diseases. However, the annotation of WM tracts is time-consuming and needs experienced neuroanatomists. In this study, to explore tract segmentation in the challenging setting of minimal annotations, we propose a novel framework utilizing only one annotated subject (subject-level one-shot) for tract segmentation. Our method is constructed by proposed registration-based peak augmentation (RPA) and uncertainty-based refining (URe) modules. RP…
Deep Learning-based 2D/3D registration enables fast, robust, and accurate X-ray to CT image fusion when large annotated paired datasets are available for training. However, the need for paired CT volume and X-ray images with ground truth registration limits the applicability in interventional scenarios. An alternative is to use simulated X-ray projections from CT volumes, thus removing the need for paired annotated datasets. Deep Neural Networks trained exclusively on simulated X-ray projections can perform significantly worse on real X-ray images due to the domain gap. We propose a self-super…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sichuan Cancer Hospital and Institute（四川省肿瘤医院（四川省肿瘤研究所））；Sichuan Cancer Center（四川省癌症中心）；Sichuan University（四川大学）；West China Hospital（华西医院）；Johns Hopkins University（约翰斯·霍普金斯大学）；SenseTime Research（商汤科技研究院）；Rutgers University（罗格斯大学）
Whole abdominal organ segmentation is important in diagnosing abdomen lesions, radiotherapy, and follow-up. However, oncologists' delineating all abdominal organs from 3D volumes is time-consuming and very expensive. Deep learning-based medical image segmentation has shown the potential to reduce manual delineation efforts, but it still requires a large-scale fine annotated dataset for training, and there is a lack of large-scale datasets covering the whole abdomen region with accurate and detailed annotations for the whole abdominal organ segmentation. In this work, we establish a new large-s…
Image registration is one of the most challenging problems in medical image analysis. In the recent years, deep learning based approaches became quite popular, providing fast and performing registration strategies. In this short paper, we summarise our work presented on Learn2Reg challenge 2020. The main contributions of our work rely on (i) a symmetric formulation, predicting the transformations from source to target and from target to source simultaneously, enforcing the trained representations to be similar and (ii) integration of variety of publicly available datasets used both for pretrai…
University of Westminster（威斯敏斯特大学）；Calico Life Sciences LLC（Calico生命科学有限责任公司）
An end-to-end image analysis pipeline is presented for the abdominal MRI protocol used in the UK Biobank on the first 38,971 participants. Emphasis is on the processing steps necessary to ensure a high-level of data quality and consistency is produced in order to prepare the datasets for downstream quantitative analysis, such as segmentation and parameter estimation. Quality control procedures have been incorporated to detect and, where possible, correct issues in the raw data. Detection of fat-water swaps in the Dixon series is performed by a deep learning model and corrected automatically. B…
Robust deformable image registration using cycle-consistent implicit representations
Amsterdam UMC location University of Amsterdam（阿姆斯特丹大学医学中心（阿姆斯特丹大学））；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）；Amsterdam Gastroenterology Endocrinology Metabolism（阿姆斯特丹胃肠病学与内分泌代谢研究所）；Cancer Center Amsterdam（阿姆斯特丹癌症中心）
Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Imperial College London（伦敦帝国理工学院）；University of Pennsylvania（宾夕法尼亚大学）；National University of Singapore（新加坡国立大学）
GE Healthcare（通用电气医疗）；MAP5, Universite de Paris（巴黎大学MAP5实验室）
Automatic annotation of anatomical structures can help simplify workflow during interventions in numerous clinical applications but usually involves a large amount of annotated data. The complexity of the labeling task, together with the lack of representative data, slows down the development of robust solutions. In this paper, we propose a solution requiring very few annotated cases to label 3D pelvic arterial trees of patients with benign prostatic hyperplasia. We take advantage of Large Deformation Diffeomorphic Metric Mapping (LDDMM) to perform registration based on meaningful deformations…
TRUSTED: The Paired 3D Transabdominal Ultrasound and CT Human Data for Kidney Segmentation and Registration Research
University of Strasbourg（斯特拉斯堡大学）；Research Institute against Digestive Cancer (IRCAD)（消化癌症研究所（IRCAD））；Conservatoire National des Arts et Métiers (CNAM)（法国国立工艺学院（CNAM））；Sorbonne University（索邦大学）；Clinique Sainte-Anne（圣安妮诊所）；Clinique Saint Augustin（圣奥古斯丁诊所）；Institute of Image-Guided Surgery (IHU)（图像引导手术研究所（IHU））
Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?
Department of Electrical and Computer Engineering, Duke University（电子工程与计算机科学系，杜克大学）；Department of Biostatistics and Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Departments of Biostatistics and Bioinformatics, Radiology, Electrical and Computer Engineering, and Computer Science, Duke University（生物统计学与生物信息学系、放射学、电子工程与计算机科学系，杜克大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；PAII Inc.（PAII公司）；Ping An Technology（平安科技）；The First Affiliated Hospital Zhejiang University（浙江大学医学院附属第一医院）
In this work, we introduce a fast and accurate method for unsupervised 3D medical image registration. This work is built on top of a recent algorithm SAM, which is capable of computing dense anatomical/semantic correspondences between two images at the pixel level. Our method is named SAME, which breaks down image registration into three steps: affine transformation, coarse deformation, and deep deformable registration. Using SAM embeddings, we enhance these steps by finding more coherent correspondences, and providing features and a loss function with better semantic guidance. We collect a mu…
The Kennedy Institute of Rheumatology, University of Oxford（牛津大学肯尼迪风湿病研究所）；Big Data Institute, University of Oxford（牛津大学大数据研究所）；Department of Computer Science, University of Oxford（牛津大学计算机科学系）；Department of Surgery and Cancer, Imperial College London（帝国理工学院外科与癌症系）；Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康系）
StainBridge: Stain-Aware Pairwise Registration of Serial Renal Biopsy Whole-Slide Images Across Structural and Immunohistochemical Stains
University of California, Los Angeles（加州大学洛杉矶分校）；Massachusetts General Brigham（麻省总医院布里格姆）；Vanderbilt University（范德堡大学）；University of Regensburg（雷根斯堡大学）；Weill Medical College of Cornell University（康奈尔大学韦尔医学学院）；Cornell Tech（康奈尔科技）；Northwell Health（北岸医疗）
PAII Inc.（PAII公司）；Johns Hopkins University（约翰斯·霍普金斯大学）；Vanderbilt University（范德堡大学）；Chang Gung Memorial Hospital（长庚纪念医院）；Ping An Technology（平安科技）
Multi-modal image registration is a challenging problem that is also an important clinical task for many real applications and scenarios. As a first step in analysis, deformable registration among different image modalities is often required in order to provide complementary visual information. During registration, semantic information is key to match homologous points and pixels. Nevertheless, many conventional registration methods are incapable in capturing high-level semantic anatomical dense correspondences. In this work, we propose a novel multi-task learning system, JSSR, based on an end…
Leiden University Medical Center（莱顿大学医学中心）；Ankara University（安卡拉大学）；Imperial College（帝国理工学院）；Delft University of Technology（代尔夫特理工大学）
Predicting registration error can be useful for evaluation of registration procedures, which is important for the adoption of registration techniques in the clinic. In addition, quantitative error prediction can be helpful in improving the registration quality. The task of predicting registration error is demanding due to the lack of a ground truth in medical images. This paper proposes a new automatic method to predict the registration error in a quantitative manner, and is applied to chest CT scans. A random regression forest is utilized to predict the registration error locally. The forest …
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁大学医学院）；University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Chinese Academy of Sciences（中国科学院）
Multimodal image registration has many applications in diagnostic medical imaging and image-guided interventions, such as Transcatheter Arterial Chemoembolization (TACE) of liver cancer guided by intraprocedural CBCT and pre-operative MR. The ability to register peri-procedurally acquired diagnostic images into the intraprocedural environment can potentially improve the intra-procedural tumor targeting, which will significantly improve therapeutic outcomes. However, the intra-procedural CBCT often suffers from suboptimal image quality due to lack of signal calibration for Hounsfield unit, limi…
Institute of Medical Informatics, Universität zu Lübeck（吕贝克大学医学信息学研究所）
Multimodal image registration is a very challenging problem for deep learning approaches. Most current work focuses on either supervised learning that requires labelled training scans and may yield models that bias towards annotated structures or unsupervised approaches that are based on hand-crafted similarity metrics and may therefore not outperform their classical non-trained counterparts. We believe that unsupervised domain adaptation can be beneficial in overcoming the current limitations for multimodal registration, where good metrics are hard to define. Domain adaptation has so far been…
Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography
City University of Hong Kong（香港城市大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of California, San Francisco（加州大学旧金山分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Chinese University of Hong Kong（香港中文大学）；NVIDIA（英伟达）
Faculty of Computer Science Universitas Indonesia（印度尼西亚大学计算机科学学院）
The number of fetal-neonatal death in Indonesia is still high compared to developed countries. This is caused by the absence of maternal monitoring during pregnancy. This paper presents an automated measurement for fetal head circumference (HC) and abdominal circumference (AC) from the ultrasonography (USG) image. This automated measurement is beneficial to detect early fetal abnormalities during the pregnancy period. We used the convolutional neural network (CNN) method, to preprocess the USG data. After that, we approximate the head and abdominal circumference using the Hough transform algor…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Auris Health, Inc.（奥里斯健康公司）；Johns Hopkins Medicine（约翰斯·霍普金斯医学院）；Nara Institute of Science and Technology（奈良科学技术研究所）；University of Texas（德克萨斯大学）；Texas Orthopedics（德克萨斯骨科医院）
Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. We propose a method for fully automatic registration using annotations produced by a neural network. Neural networks are trained to simultaneously segment anatomy and ide…
A novel framework for fully-automated co-registration of intravascular ultrasound and optical coherence tomography imaging data
Barts Health NHS Trust（巴茨健康国民医疗服务信托）；Huazhong University of Science and Technology（华中科技大学）；Queen Mary, University of London（伦敦玛丽女王大学）；Queen Mary University London（伦敦玛丽女王大学）；University College London（伦敦大学学院）；Leiden University Medical Center（莱顿大学医学中心）；University of Bern（伯尔尼大学）
University of Science and Technology of China（中国科学技术大学）；Guangzhou First People’s Hospital（广州市第一人民医院）；South China University of Technology（华南理工大学）
In the paper, we present an approach for learning a single model that universally segments 33 anatomical structures, including vertebrae, pelvic bones, and abdominal organs. Our model building has to address the following challenges. Firstly, while it is ideal to learn such a model from a large-scale, fully-annotated dataset, it is practically hard to curate such a dataset. Thus, we resort to learn from a union of multiple datasets, with each dataset containing the images that are partially labeled. Secondly, along the line of partial labelling, we contribute an open-source, large-scale verteb…
The analysis of FFPE tissue sections stained with haematoxylin and eosin (H&E) or immunohistochemistry (IHC) is an essential part of the pathologic assessment of surgically resected breast cancer specimens. IHC staining has been broadly adopted into diagnostic guidelines and routine workflows to manually assess status and scoring of several established biomarkers, including ER, PGR, HER2 and KI67. However, this is a task that can also be facilitated by computational pathology image analysis methods. The research in computational pathology has recently made numerous substantial advances, often …
Automatic brain tumor segmentation in 2D intra-operative ultrasound images using magnetic resonance imaging tumor annotations
Department of Health Research（健康研究系）；SINTEF Digital（SINTEF数字）；Department of Physics（物理系）；Norwegian University of Science and Technology (NTNU)（挪威科学与技术大学（NTNU））；Department of Circulation and Medical Imaging（循环与医学成像系）；St. Olavs hospital（圣奥拉夫医院）；Department of Neuromedicine and Movement Science（神经医学与运动科学系）
++nnU-Net: Scaling nnU-Net with Prefix-Based Data Augmentation
Center Algoritmi / LASI, University of Minho, Braga, Portugal（阿尔戈里米中心/拉斯伊大学，明霍大学，布拉加，葡萄牙）；Institute for Artificial Intelligence in Medicine, University Medicine Essen, Essen, Germany（医学人工智能研究所，埃森医学院，埃森，德国）；Institute of Medical Informatics / Dept. of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Germany（医学信息学研究所/口腔和颅面外科部，亚琛大学医院，德国）；Faculty of Computer Science, University of Duisburg-Essen, Essen, Germany（计算机科学学院，杜伊斯堡-埃森大学，埃森，德国）
In the field of medical image, deep convolutional neural networks(ConvNets) have achieved great success in the classification, segmentation, and registration tasks thanks to their unparalleled capacity to learn image features. However, these tasks often require large amounts of manually annotated data and are labor-intensive. Therefore, it is of significant importance for us to study unsupervised semantic feature learning tasks. In our work, we propose to learn features in medical images by training ConvNets to recognize the geometric transformation applied to images and present a simple self-…
