University of Warwick（华威大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）；The Alan Turing Institute（艾伦·图灵研究所）；University Hospitals Coventry & Warwickshire（考文垂与沃里克郡大学医院）
Ruqayya Awan, Shan E Ahmed Raza, Johannes Lotz, Nick Weiss, Nasir Rajpoot
Cross-slide image analysis provides additional information by analysing the expression of different biomarkers as compared to a single slide analysis. These biomarker stained slides are analysed side by side, revealing unknown relations between them. During the slide preparation, a tissue section may be placed at an arbitrary orientation as compared to other sections of the same tissue block. The problem is compounded by the fact that tissue contents are likely to change from one section to the next and there may be unique artefacts on some of the slides. This makes registration of each section to a reference section of the same tissue block an important pre-requisite task before any cross-slide analysis. We propose a deep feature based registration (DFBR) method which utilises data-driven features to estimate the rigid transformation. We adopted a multi-stage strategy for improving the quality of registration. We also developed a visualisation tool to view registered pairs of WSIs at different magnifications. With the help of this tool, one can apply a transformation on the fly without the need to generate transformed source WSI in a pyramidal form. We compared the performance of data-driven features with that of hand-crafted features on the COMET dataset. Our approach can align the images with low registration errors. Generally, the success of non-rigid registration is dependent on the quality of rigid registration. To evaluate the efficacy of the DFBR method, the first two steps of the ANHIR winner's framework are replaced with our DFBR to register challenge provided image pairs. The modified framework produces comparable results to that of challenge winning team.
Muroran Institute of Technology（室兰工业大学）；Niigata University of Health and Welfare（新潟医疗福祉大学）；Konica Minolta, Inc.（柯尼卡美能达公司）
Multi-stain whole-slide-image (WSI) registration is an active field of research. It is unclear, however, how the current WSI registration methods would perform on a real-world data set. AutomatiC Registration Of Breast cAncer Tissue (ACROBAT) challenge is held to verify the performance of the current WSI registration methods by using a new dataset that originates from routine diagnostics to assess real-world applicability. In this report, we present our solution for the ACROBAT challenge. We employ a two-step approach including rigid and non-rigid transforms. The experimental results show that…
STAR: A Fast and Robust Rigid Registration Framework for Serial Histopathological Images
School of Biological Science and Medical Engineering, Beihang University, Beijing, China（生物科学与医学工程学院，北航，北京，中国）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北航，北京，中国）
Joint analysis of multiple biomarker images and tissue morphology is important for disease diagnosis, treatment planning and drug development. It requires cross-staining comparison among Whole Slide Images (WSIs) of immuno-histochemical and hematoxylin and eosin (H&E) microscopic slides. However, automatic, and fast cross-staining alignment of enormous gigapixel WSIs at single-cell precision is challenging. In addition to morphological deformations introduced during slide preparation, there are large variations in cell appearance and tissue morphology across different staining. In this paper, …
2D-3D Deformable Image Registration of Histology Slide and Micro-CT with ML-based Initialization
ImFusion GmbH（ImFusion有限公司）；Technical University of Munich（慕尼黑工业大学）；Institute of Pathology, Heinrich Heine University and University Hospital of Düsseldorf（海因里希·海涅大学与杜塞尔多夫大学医院病理研究所）；Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut（弗劳恩霍夫高速动力学研究所（恩斯特·马赫研究所））
RegWSI: Whole Slide Image Registration using Combined Deep Feature- and Intensity-Based Methods: Winner of the ACROBAT 2023 Challenge
University of Applied Sciences Western Switzerland（瑞士西应用科学大学）；AGH University of Kraków（克拉科夫AGH大学）；University of Padova（帕多瓦大学）；University of Geneva（日内瓦大学）
Information Systems Institute, University of Applied Sciences Western Switzerland（西瑞士应用科学大学信息系统学院）；AGH University of Kraków（克拉科夫AGH大学）；University of Padova（帕多瓦大学）；University of Geneva（日内瓦大学）
Imaging Informatics Division（成像信息学部门）；Bioinformatics Institute（生物信息学研究所）；A*STAR（新加坡科技研究局）；Lucence Diagnostics（Lucence诊断公司）；Institute of Bioengineering and Nanotechnology（生物工程与纳米技术研究所）；Singapore General Hospital（新加坡综合医院）；National Cancer Centre Singapore（新加坡国家癌症中心）；Institute of Molecular and Cell Biology（分子与细胞生物学研究所）；National University of Singapore（新加坡国立大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Image and Pervasive Access Lab（图像与普适访问实验室）
Motivation: High resolution 2D whole slide imaging provides rich information about the tissue structure. This information can be a lot richer if these 2D images can be stacked into a 3D tissue volume. A 3D analysis, however, requires accurate reconstruction of the tissue volume from the 2D image stack. This task is not trivial due to the distortions that each individual tissue slice experiences while cutting and mounting the tissue on the glass slide. Performing registration for the whole tissue slices may be adversely affected by the deformed tissue regions. Consequently, regional registratio…
UWarp: A Whole Slide Image Registration Pipeline to Characterize Scanner-Induced Local Domain Shift
Ummon HealthTech SAS；Centre Georges-François Leclerc（乔治-弗朗索瓦·勒克莱尔中心）；ICMUB laboratory, Université de Bourgogne Europe（勃艮第欧洲大学ICMUB实验室）；University Hospital of Dijon（第戎大学医院）；Hospital de Tortosa Verge de la Cinta (HTVC), Catalan Health Institute (ICS)（托尔托萨维尔赫-德拉辛塔医院 (HTVC)，加泰罗尼亚卫生研究所 (ICS)）；Pere Virgili Institute for Health Research (IISPV)（佩雷·维尔吉利健康研究所 (IISPV)）；Universitat Rovira i Virgili (URV)（罗维拉-威尔吉利大学 (URV)）；Nałęcz Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences（波兰科学院纳文奇生物控制论与生物医学工程研究所）；Cairo University（开罗大学）；University of Udine（乌迪内大学）
Variously stained histology slices are routinely used by pathologists to assess extracted tissue samples from various anatomical sites and determine the presence or extent of a disease. Evaluation of sequential slides is expected to enable a better understanding of the spatial arrangement and growth patterns of cells and vessels. In this paper we present a practical two-step approach based on diffeomorphic registration to align digitized sequential histopathology stained slides to each other, starting with an initial affine step followed by the estimation of a detailed deformation field.
Radboud University Medical Center（拉德堡德大学医学中心）；Canisius-Wilhelmina Hospital（卡尼修斯-威廉明娜医院）；Jeroen Bosch Hospital（耶伦博斯医院）
Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by: (1) noisy and expensive reference standards established by pathologists, (2) lack of generalization due to staining variation across laboratories, and (3) high compu…
CT-Guided Spatially-varying Regularization for Voxel-Wise Deformable Whole-Body PET Registration
Institute of Medical Technology, Peking University Health Science Center（北京大学医学部医学技术研究所）；Ren Ji Hospital, Shanghai Jiao Tong University（上海交通大学仁济医院）；University of Oxford（牛津大学）
StainBridge: Stain-Aware Pairwise Registration of Serial Renal Biopsy Whole-Slide Images Across Structural and Immunohistochemical Stains
University of California, Los Angeles（加州大学洛杉矶分校）；Massachusetts General Brigham（麻省总医院布里格姆）；Vanderbilt University（范德堡大学）；University of Regensburg（雷根斯堡大学）；Weill Medical College of Cornell University（康奈尔大学韦尔医学学院）；Cornell Tech（康奈尔科技）；Northwell Health（北岸医疗）
The application of deep learning to pathology assumes the existence of digital whole slide images of pathology slides. However, slide digitization is bottlenecked by the high cost of precise motor stages in slide scanners that are needed for position information used for slide stitching. We propose GloFlow, a two-stage method for creating a whole slide image using optical flow-based image registration with global alignment using a computationally tractable graph-pruning approach. In the first stage, we train an optical flow predictor to predict pairwise translations between successive video fr…
The analysis of FFPE tissue sections stained with haematoxylin and eosin (H&E) or immunohistochemistry (IHC) is an essential part of the pathologic assessment of surgically resected breast cancer specimens. IHC staining has been broadly adopted into diagnostic guidelines and routine workflows to manually assess status and scoring of several established biomarkers, including ER, PGR, HER2 and KI67. However, this is a task that can also be facilitated by computational pathology image analysis methods. The research in computational pathology has recently made numerous substantial advances, often …
Unsupervised Deformable Image Registration with Local-Global Attention and Image Decomposition
Institute of Medical Technology, Peking University Health Science Center（北京大学医学部医学技术研究所）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）；Department of Radiology, Peking University Third Hospital（北京大学第三医院放射科）；Department of Pediatrics, Peking University First Hospital（北京大学第一医院儿科）；Pediatric Epilepsy Center, Peking University First Hospital（北京大学第一医院儿童癫痫中心）；School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）；NVIDIA Corporation（英伟达公司）
Transformers have recently shown promise for medical image applications, leading to an increasing interest in developing such models for medical image registration. Recent advancements in designing registration Transformers have focused on using cross-attention (CA) to enable a more precise understanding of spatial correspondences between moving and fixed images. Here, we propose a novel CA mechanism that computes windowed attention using deformable windows. In contrast to existing CA mechanisms that require intensive computational complexity by either computing CA globally or locally with a f…
University College London（伦敦大学学院）；InstaDeep Co.（InstaDeep公司）；City University of Hong Kong（香港城市大学）
In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those being registered. As an example, we focus on aligning intra-subject multiparametric Magnetic Resonance (mpMR) images, between T2-weighted (T2w) scans and diffusion-weighted scans with high b-value (DWI$_{high-b}$). For the application of localising tumours in mpMR images, diffusion scans with zero b-value (DWI$_{b=0}$) are considered easier to register to T2w due to the availa…
The ACROBAT 2022 Challenge: Automatic Registration Of Breast Cancer Tissue
Karolinska Institutet（卡罗林斯卡学院）；University of Turku（图尔库大学）；University of Eastern Finland（东芬兰大学）；Zealand University Hospital（西兰大学医院）；Gestalt Diagnostics（格式塔诊断公司）；Moffitt Cancer Center（莫菲特癌症中心）；University of Applied Sciences Western Switzerland（瑞士西部应用科学大学）；AGH University of Kraków（克拉科夫AGH科技大学）；University of Geneva（日内瓦大学）；University of Padova（帕多瓦大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）；University of Twente（特温特大学）；Muroran Institute of Technology（室兰工业大学）；Niigata University of Health and Welfare（新潟医疗福祉大学）；Konica Minolta, Inc.（柯尼卡美能达株式会社）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；Karolinska University Hospital（卡罗林斯卡大学医院）；Tampere University（坦佩雷大学）
A solution for co-locating 2D histology images in 3D for histology-to-CT and MR image registration: closing the loop for bone sarcoma treatment planning
Multimodal Cross-Task Interaction for Survival Analysis in Whole Slide Pathological Images
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Science, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）理学院）
Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）；Leiden University Medical Center（莱顿大学医学中心）
Recently, joint registration and segmentation has been formulated in a deep learning setting, by the definition of joint loss functions. In this work, we investigate joining these tasks at the architectural level. We propose a registration network that integrates segmentation propagation between images, and a segmentation network to predict the segmentation directly. These networks are connected into a single joint architecture via so-called cross-stitch units, allowing information to be exchanged between the tasks in a learnable manner. The proposed method is evaluated in the context of adapt…
Single-photon emission computed tomography (SPECT) is a widely applied imaging approach for diagnosis of coronary artery diseases. Attenuation maps (u-maps) derived from computed tomography (CT) are utilized for attenuation correction (AC) to improve diagnostic accuracy of cardiac SPECT. However, SPECT and CT are obtained sequentially in clinical practice, which potentially induces misregistration between the two scans. Convolutional neural networks (CNN) are powerful tools for medical image registration. Previous CNN-based methods for cross-modality registration either directly concatenated t…
University of Oxford（牛津大学）；University College London（伦敦大学学院）；InstaDeep（InstaDeep公司）；City University of Hong Kong（香港城市大学）；Radboud University Nijmegen Medical Centre（拉德堡德大学奈梅亨医学中心）
The ability to adapt medical image segmentation networks for a novel class such as an unseen anatomical or pathological structure, when only a few labelled examples of this class are available from local healthcare providers, is sought-after. This potentially addresses two widely recognised limitations in deploying modern deep learning models to clinical practice, expertise-and-labour-intensive labelling and cross-institution generalisation. This work presents the first 3D few-shot interclass segmentation network for medical images, using a labelled multi-institution dataset from prostate canc…
Multi-atlas segmentation (MAS) is a promising framework for medical image segmentation. Generally, MAS methods register multiple atlases, i.e., medical images with corresponding labels, to a target image; and the transformed atlas labels can be combined to generate target segmentation via label fusion schemes. Many conventional MAS methods employed the atlases from the same modality as the target image. However, the number of atlases with the same modality may be limited or even missing in many clinical applications. Besides, conventional MAS methods suffer from the computational burden of reg…
University of Cambridge（剑桥大学）；Cancer Research UK Cambridge Centre（英国癌症研究中心剑桥中心）；Medical University Vienna（维也纳医科大学）；Cambridge University Hospitals National Health Service (NHS) Foundation Trust（剑桥大学医院国民保健服务基金会信托）；Eindhoven University of Technology（埃因霍温理工大学）；Leiden University（莱顿大学）；University College London（伦敦大学学院）；Cancer Research UK Cambridge Institute（英国癌症研究中心剑桥研究所）
This study investigates the use of the unsupervised deep learning framework VoxelMorph for deformable registration of longitudinal abdominopelvic CT images acquired in patients with bone metastases from breast cancer. The CT images were refined prior to registration by automatically removing the CT table and all other extra-corporeal components. To improve the learning capabilities of VoxelMorph when only a limited amount of training data is available, a novel incremental training strategy is proposed based on simulated deformations of consecutive CT images. In a 4-fold cross-validation scheme…
University of Leeds（利兹大学）；Leeds Teaching Hospitals NHS Trust（利兹教学医院NHS信托基金会）
Weakly-supervised classification of histopathology slides is a computationally intensive task, with a typical whole slide image (WSI) containing billions of pixels to process. We propose Discriminative Region Active Sampling for Multiple Instance Learning (DRAS-MIL), a computationally efficient slide classification method using attention scores to focus sampling on highly discriminative regions. We apply this to the diagnosis of ovarian cancer histological subtypes, which is an essential part of the patient care pathway as different subtypes have different genetic and molecular profiles, treat…
UDCR: Unsupervised Aortic DSA/CTA Rigid Registration Using Deep Reinforcement Learning and Overlap Degree Calculation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Shanghai Changhai Hospital, Naval Medical University（海军军医大学上海长海医院）；Dalian University of Technology（大连理工大学）；School of Computer Science and Information Security, Guilin University of Electronic Technology（桂林电子科技大学计算机科学与信息安全学院）；The Fourth Hospital of Hebei Medical University and Hebei Tumor Hospital（河北医科大学第四医院暨河北省肿瘤医院）；Erasmus MC, University Medical Center Rotterdam（鹿特丹伊拉斯姆斯大学医学中心）
In recent years, the availability of digitized Whole Slide Images (WSIs) has enabled the use of deep learning-based computer vision techniques for automated disease diagnosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized ($\sim$100K pixels), making them infeasible to be used directly for training deep neural networks. Also, often only slide-level labels are available for training as detailed annotations are tedious and can be time-consuming for experts. Approaches using multiple-instance learning (MIL) frameworks have been shown to overcome th…
ShanghaiTech University（上海科技大学）；University of Zürich（苏黎世大学）；ETH Zürich（苏黎世联邦理工学院）
Multi-spectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) that offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of multi-modal images remains challenging, chiefly due to the entirely different image contrast rendered by these modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation …
Huazhong University of Science and Technology（华中科技大学）；Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics（武汉光电国家研究中心Britton Chance生物医学光子学研究中心）；Massachusetts Institute of Technology（麻省理工学院）；Shenzhen Huazhong University of Science and Technology Research Institute（深圳华中科技大学研究院）
We combine generative adversarial network (GAN) with light microscopy to achieve deep learning super-resolution under a large field of view (FOV). By appropriately adopting prior microscopy data in an adversarial training, the neural network can recover a high-resolution, accurate image of new specimen from its single low-resolution measurement. Its capacity has been broadly demonstrated via imaging various types of samples, such as USAF resolution target, human pathological slides, fluorescence-labelled fibroblast cells, and deep tissues in transgenic mouse brain, by both wide-field and light…
Magnetic resonance imaging (MRI) has great potential to improve prostate cancer diagnosis. It can spare men with a normal exam from undergoing invasive biopsy while making biopsies more accurate in men with lesions suspicious for cancer. Yet, the subtle differences between cancer and confounding conditions, render the interpretation of MRI challenging. The tissue collected from patients that undergo pre-surgical MRI and radical prostatectomy provides a unique opportunity to correlate histopathology images of the entire prostate with MRI in order to accurately map the extent of prostate cancer …
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
