Recently, joint registration and segmentation has been formulated in a deep learning setting, by the definition of joint loss functions. In this work, we investigate joining these tasks at the architectural level. We propose a registration network that integrates segmentation propagation between images, and a segmentation network to predict the segmentation directly. These networks are connected into a single joint architecture via so-called cross-stitch units, allowing information to be exchanged between the tasks in a learnable manner. The proposed method is evaluated in the context of adaptive image-guided radiotherapy, using daily prostate CT imaging. Two datasets from different institutes and manufacturers were involved in the study. The first dataset was used for training (12 patients) and validation (6 patients), while the second dataset was used as an independent test set (14 patients). In terms of mean surface distance, our approach achieved $1.06 \pm 0.3$ mm, $0.91 \pm 0.4$ mm, $1.27 \pm 0.4$ mm, and $1.76 \pm 0.8$ mm on the validation set and $1.82 \pm 2.4$ mm, $2.45 \pm 2.4$ mm, $2.45 \pm 5.0$ mm, and $2.57 \pm 2.3$ mm on the test set for the prostate, bladder, seminal vesicles, and rectum, respectively. The proposed multi-task network outperformed single-task networks, as well as a network only joined through the loss function, thus demonstrating the capability to leverage the individual strengths of the segmentation and registration tasks. The obtained performance as well as the inference speed make this a promising candidate for daily re-contouring in adaptive radiotherapy, potentially reducing treatment-related side effects and improving quality-of-life after treatment.
Leiden University Medical Center（莱顿大学医学中心）；University Medical Center Utrecht（乌得勒支大学医学中心）
Joint image registration and segmentation has long been an active area of research in medical imaging. Here, we reformulate this problem in a deep learning setting using adversarial learning. We consider the case in which fixed and moving images as well as their segmentations are available for training, while segmentations are not available during testing; a common scenario in radiotherapy. The proposed framework consists of a 3D end-to-end generator network that estimates the deformation vector field (DVF) between fixed and moving images in an unsupervised fashion and applies this DVF to the …
Cervical cancer threatens the health of women seriously. Radiotherapy is one of the main therapy methods but with high risk of acute hematologic toxicity. Delineating the bone marrow (BM) for sparing using computer tomography (CT) images to plan before radiotherapy can effectively avoid this risk. Comparing with magnetic resonance (MR) images, CT lacks the ability to express the activity of BM. Thus, in current clinical practice, medical practitioners manually delineate the BM on CT images by corresponding to MR images. However, the time?consuming delineating BM by hand cannot guarantee the ac…
Leiden University Medical Center（莱顿大学医学中心）；Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）
Medical image registration and segmentation are two of the most frequent tasks in medical image analysis. As these tasks are complementary and correlated, it would be beneficial to apply them simultaneously in a joint manner. In this paper, we formulate registration and segmentation as a joint problem via a Multi-Task Learning (MTL) setting, allowing these tasks to leverage their strengths and mitigate their weaknesses through the sharing of beneficial information. We propose to merge these tasks not only on the loss level, but on the architectural level as well. We studied this approach in th…
Adaptive radiotherapy (ART), especially online ART, effectively accounts for positioning errors and anatomical changes. One key component of online ART is accurately and efficiently delineating organs at risk (OARs) and targets on online images, such as CBCT, to meet the online demands of plan evaluation and adaptation. Deep learning (DL)-based automatic segmentation has gained great success in segmenting planning CT, but its applications to CBCT yielded inferior results due to the low image quality and limited available contour labels for training. To overcome these obstacles to online CBCT s…
DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapy
Department of Computer Science and Software Engineering, Concordia University（计算机科学与软件工程系，康科迪亚大学）；Department of Electrical and Computer Engineering, Concordia University（电气与计算机工程系，康科迪亚大学）；Elekta Ltd.（Elekta公司）；Diagnos Medical Systems（Diagnos医疗系统）
Université de Bourgogne Franche-Comté（勃艮第-弗朗什孔泰大学）；CGFL（CGFL（第戎乔治·弗朗索瓦·勒克莱尔中心））；University Hospital of Dijon（第戎大学医院）
Deep learning has shown unprecedented success in a variety of applications, such as computer vision and medical image analysis. However, there is still potential to improve segmentation in multimodal images by embedding prior knowledge via learning-based shape modeling and registration to learn the modality invariant anatomical structure of organs. For example, in radiotherapy automatic prostate segmentation is essential in prostate cancer diagnosis, therapy, and post-therapy assessment from T2-weighted MR or CT images. In this paper, we present a fully automatic deep generative model-driven m…
OncoReg: Medical Image Registration for Oncological Challenges
Institute of Medical Informatics, University of Lübeck（吕贝克大学医学信息学研究所）；Institute of Radiology and Nuclear Medicine, University Hospital Schleswig-Holstein（石勒斯维希-霍尔斯坦大学医院放射科和核医学研究所）；Department of Biomedical Engineering and Center for Biotechnology and Interdisciplinary Studies, Rensselaer Polytechnic Institute（伦塞拉塞尔理工学院生物医学工程系和生物技术与跨学科研究中心）；School of Computer Science, University of Birmingham（伯明翰大学计算机科学学院）；Division of Informatics, Imaging and Data Sciences, University of Manchester（曼彻斯特大学信息学、成像和数据科学系）；DAMO Academy, Alibaba Group（阿里集团DAMO学院）；Hangzhou Shengshi Technology Co., Ltd（杭州盛世科技有限公司）；Department of Radiation Oncology, University Hospital Schleswig-Holstein（石勒斯维希-霍尔斯坦大学医院放射肿瘤科）；EchoScout GmbH；Radboud University Medical Center, Nijmegen（奈密根大学医学中心）；Institute of Interventional Radiology, University Hospital Schleswig-Holstein（石勒斯维希-霍尔斯坦大学医院介入放射科）
Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；Graduate School of Medicine, Kyoto University（京都大学医学研究科）
Organ shape reconstruction based on a single-projection image during treatment has wide clinical scope, e.g., in image-guided radiotherapy and surgical guidance. We propose an image-to-graph convolutional network that achieves deformable registration of a 3D organ mesh for a single-viewpoint 2D projection image. This framework enables simultaneous training of two types of transformation: from the 2D projection image to a displacement map, and from the sampled per-vertex feature to a 3D displacement that satisfies the geometrical constraint of the mesh structure. Assuming application to radiati…
FAU Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；University of Leeds（利兹大学）
Comparison of microvascular circulation on fundoscopic images is a non-invasive clinical indication for the diagnosis and monitoring of diseases, such as diabetes and hypertensions. The differences between intra-patient images can be assessed quantitatively by registering serial acquisitions. Due to the variability of the images (i.e. contrast, luminosity) and the anatomical changes of the retina, the registration of fundus images remains a challenging task. Recently, several deep learning approaches have been proposed to register fundus images in an end-to-end fashion, achieving remarkable re…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）
Image-guided adaptive lung radiotherapy requires accurate tumor and organs segmentation from during treatment cone-beam CT (CBCT) images. Thoracic CBCTs are hard to segment because of low soft-tissue contrast, imaging artifacts, respiratory motion, and large treatment induced intra-thoracic anatomic changes. Hence, we developed a novel Patient-specific Anatomic Context and Shape prior or PACS-aware 3D recurrent registration-segmentation network for longitudinal thoracic CBCT segmentation. Segmentation and registration networks were concurrently trained in an end-to-end framework and implemente…
University of Edinburgh（爱丁堡大学）；Heriot-Watt University（赫瑞-瓦特大学）
Autofluorescence lifetime images reveal unique characteristics of endogenous fluorescence in biological samples. Comprehensive understanding and clinical diagnosis rely on co-registration with the gold standard, histology images, which is extremely challenging due to the difference of both images. Here, we show an unsupervised image-to-image translation network that significantly improves the success of the co-registration using a conventional optimisation-based regression network, applicable to autofluorescence lifetime images at different emission wavelengths. A preliminary blind comparison …
SynthRAD2025 Grand Challenge dataset: generating synthetic CTs for radiotherapy
LMU University Hospital, LMU Munich（慕尼黑大学附属医院，慕尼黑大学）；Radboud University Medical Center（拉德堡德大学医学中心）；University Medical Center Groningen, University of Groningen（格罗宁根大学医学中心，格罗宁根大学）；University Hospital of Cologne（科隆大学附属医院）；University Medical Center Utrecht（乌得勒支大学医学中心）；German Cancer Consortium (DKTK)（德国癌症联盟）；Bavarian Cancer Research Center (BZKF)（巴伐利亚癌症研究中心）；Skåne University Hospital（斯科讷大学医院）；Lund University（隆德大学）；Deutsches Krebsforschungszentrum (DKFZ)（德国癌症研究中心）
The University of Tokyo（东京大学）；National Institute for Quantum and Radiological Science and Technology（国立量子与放射科学技术研究所）
During the radiotherapy treatment of patients with lung cancer, the radiation delivered to healthy tissue around the tumor needs to be minimized, which is difficult because of respiratory motion and the latency of linear accelerator systems. In the proposed study, we first use the Lucas-Kanade pyramidal optical flow algorithm to perform deformable image registration of chest computed tomography scan images of four patients with lung cancer. We then track three internal points close to the lung tumor based on the previously computed deformation field and predict their position with a recurrent …
