The prediction of microsatellite instability (MSI) and microsatellite stability (MSS) is essential in predicting both the treatment response and prognosis of gastrointestinal cancer. In clinical practice, a universal MSI testing is recommended, but the accessibility of such a test is limited. Thus, a more cost-efficient and broadly accessible tool is desired to cover the traditionally untested patients. In the past few years, deep-learning-based algorithms have been proposed to predict MSI directly from haematoxylin and eosin (H&E)-stained whole-slide images (WSIs). Such algorithms can be summarized as (1) patch-level MSI/MSS prediction, and (2) patient-level aggregation. Compared with the advanced deep learning approaches that have been employed for the first stage, only the naïve first-order statistics (e.g., averaging and counting) were employed in the second stage. In this paper, we propose a simple yet broadly generalizable patient-level MSI aggregation (MAg) method to effectively integrate the precious patch-level information. Briefly, the entire probabilistic distribution in the first stage is modeled as histogram-based features to be fused as the final outcome with machine learning (e.g., SVM). The proposed MAg method can be easily used in a plug-and-play manner, which has been evaluated upon five broadly used deep neural networks: ResNet, MobileNetV2, EfficientNet, Dpn and ResNext. From the results, the proposed MAg method consistently improves the accuracy of patient-level aggregation for two publicly available datasets. It is our hope that the proposed method could potentially leverage the low-cost H&E based MSI detection method. The code of our work has been made publicly available at https://github.com/Calvin-Pang/MAg.
University of Warwick（华威大学）；University Hospitals Coventry & Warwickshire（考文垂与沃里克郡大学医院）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）；The Alan Turing Institute（艾伦·图灵研究所）
The accurate diagnosis and molecular profiling of colorectal cancers are critical for planning the best treatment options for patients. Microsatellite instability (MSI) or mismatch repair (MMR) status plays a vital role in appropriate treatment selection, has prognostic implications and is used to investigate the possibility of patients having underlying genetic disorders (Lynch syndrome). NICE recommends that all CRC patients should be offered MMR/MSI testing. Immunohistochemistry is commonly used to assess MMR status with subsequent molecular testing performed as required. This incurs signif…
Microsatellite instability (MSI) is a tumor phenotype whose diagnosis largely impacts patient care in colorectal cancers (CRC), and is associated with response to immunotherapy in all solid tumors. Deep learning models detecting MSI tumors directly from H&E stained slides have shown promise in improving diagnosis of MSI patients. Prior deep learning models for MSI detection have relied on neural networks pretrained on ImageNet dataset, which does not contain any medical image. In this study, we leverage recent advances in self-supervised learning by training neural networks on histology images…
DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
Netherlands Cancer Institute（荷兰癌症研究所）；University of Amsterdam（阿姆斯特丹大学）；Ellogon AI B.V.（Ellogon人工智能有限公司）；Radboud University Medical Center（拉德堡德大学医学中心）
Siemens Healthineers（西门子医疗）；University Medical Center Goettingen（哥廷根大学医学中心）
Current approaches for classification of whole slide images (WSI) in digital pathology predominantly utilize a two-stage learning pipeline. The first stage identifies areas of interest (e.g. tumor tissue), while the second stage processes cropped tiles from these areas in a supervised fashion. During inference, a large number of tiles are combined into a unified prediction for the entire slide. A major drawback of such approaches is the requirement for task-specific auxiliary labels which are not acquired in clinical routine. We propose a novel learning pipeline for WSI classification that is …
University of Warwick（华威大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学MEVIS研究所）；The Alan Turing Institute（艾伦·图灵研究所）；University Hospitals Coventry & Warwickshire（考文垂与沃里克郡大学医院）
Cross-slide image analysis provides additional information by analysing the expression of different biomarkers as compared to a single slide analysis. These biomarker stained slides are analysed side by side, revealing unknown relations between them. During the slide preparation, a tissue section may be placed at an arbitrary orientation as compared to other sections of the same tissue block. The problem is compounded by the fact that tissue contents are likely to change from one section to the next and there may be unique artefacts on some of the slides. This makes registration of each sectio…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Weill Cornell Graduate School for Medical Sciences（威尔康奈尔医学研究生院）
Pathologic analysis of surgical excision specimens for breast carcinoma is important to evaluate the completeness of surgical excision and has implications for future treatment. This analysis is performed manually by pathologists reviewing histologic slides prepared from formalin-fixed tissue. In this paper, we present Deep Multi-Magnification Network trained by partial annotation for automated multi-class tissue segmentation by a set of patches from multiple magnifications in digitized whole slide images. Our proposed architecture with multi-encoder, multi-decoder, and multi-concatenation out…
Stony Brook University（石溪大学）；University of Utah School of Medicine（犹他大学医学院）
We study the attention of pathologists as they examine whole-slide images (WSIs) of prostate cancer tissue using a digital microscope. To the best of our knowledge, our study is the first to report in detail how pathologists navigate WSIs of prostate cancer as they accumulate information for their diagnoses. We collected slide navigation data (i.e., viewport location, magnification level, and time) from 13 pathologists in 2 groups (5 genitourinary (GU) specialists and 8 general pathologists) and generated visual attention heatmaps and scanpaths. Each pathologist examined five WSIs from the TCG…
Malaria is mosquito-borne blood disease caused by parasites of the genus Plasmodium. Conventional diagnostic tool for malaria is the examination of stained blood cell of patient in microscope. The blood to be tested is placed in a slide and is observed under a microscope to count the number of infected RBC. An expert technician is involved in the examination of the slide with intense visual and mental concentration. This is tiresome and time consuming process. In this paper, we construct a new mage processing system for detection and quantification of plasmodium parasites in blood smear slide,…
Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide Images
Center for Applied Artificial Intelligence, University of Kentucky（应用人工智能中心，肯塔基大学）；Sanders-Brown Center on Aging, University of Kentucky（老龄化桑德斯-布朗中心，肯塔基大学）；Department of Electrical and Computer Engineering, University of Kentucky（电气与计算机工程系，肯塔基大学）；Division of Neuropathology, University of Pittsburgh（神经病理学部，匹兹堡大学）；Department of Biomedical Informatics, Emory University（生物医学信息学系，埃默里大学）；Department of Pathology and Laboratory Medicine, University of California Davis（病理学与实验室医学系，加州大学戴维斯分校）；Department of Laboratory Medicine and Pathology, University of Texas Health（实验室医学与病理学系，德克萨斯大学健康科学中心）
Measuring and Predicting Where and When Pathologists Focus their Visual Attention while Grading Whole Slide Images of Cancer
Department of Computer Science（计算机科学系）；Department of Biomedical Informatics（生物医学信息学系）；Department of Psychology（心理学系）；Department of Pathology and Laboratory Medicine（病理学与实验室医学系）；Department of Pathology（病理学系）
UWarp: A Whole Slide Image Registration Pipeline to Characterize Scanner-Induced Local Domain Shift
Ummon HealthTech SAS；Centre Georges-François Leclerc（乔治-弗朗索瓦·勒克莱尔中心）；ICMUB laboratory, Université de Bourgogne Europe（勃艮第欧洲大学ICMUB实验室）；University Hospital of Dijon（第戎大学医院）；Hospital de Tortosa Verge de la Cinta (HTVC), Catalan Health Institute (ICS)（托尔托萨维尔赫-德拉辛塔医院 (HTVC)，加泰罗尼亚卫生研究所 (ICS)）；Pere Virgili Institute for Health Research (IISPV)（佩雷·维尔吉利健康研究所 (IISPV)）；Universitat Rovira i Virgili (URV)（罗维拉-威尔吉利大学 (URV)）；Nałęcz Institute of Biocybernetics and Biomedical Engineering Polish Academy of Sciences（波兰科学院纳文奇生物控制论与生物医学工程研究所）；Cairo University（开罗大学）；University of Udine（乌迪内大学）
Helmholtz Munich（亥姆霍兹慕尼黑中心）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In many histopathology tasks, sample classification depends on morphological details in tissue or single cells that are only visible at the highest magnification. For a pathologist, this implies tedious zooming in and out, while for a computational decision support algorithm, it leads to the analysis of a huge number of small image patches per whole slide image (WSI). Attention-based multiple instance learning (MIL), where attention estimation is learned in a weakly supervised manner, has been successfully applied in computational histopathology, but it is challenged by large numbers of irrele…
Image Data Analytics, Informatics IT, Merck \& Co., Inc., West Point, PA 19486, USA Image Data Analytics, Informatics IT, Merck \& Co., Inc., Rahway, NJ 07065, USA
It is common for pathologists to annotate specific regions of the tissue, such as tumor, directly on the glass slide with markers. Although this practice was helpful prior to the advent of histology whole slide digitization, it often occludes important details which are increasingly relevant to immuno-oncology due to recent advancements in digital pathology imaging techniques. The current work uses a generative adversarial network with cycle loss to remove these annotations while still maintaining the underlying structure of the tissue by solving an image-to-image translation problem. We train…
Northeastern University（东北大学）；University of the Chinese Academy of Sciences（中国科学院大学）；China Medical University（中国医科大学）
The deep neural network is a research hotspot for histopathological image analysis, which can improve the efficiency and accuracy of diagnosis for pathologists or be used for disease screening. The whole slide pathological image can reach one gigapixel and contains abundant tissue feature information, which needs to be divided into a lot of patches in the training and inference stages. This will lead to a long convergence time and large memory consumption. Furthermore, well-annotated data sets are also in short supply in the field of digital pathology. Inspired by the pathologist's clinical di…
High-performance Data Management for Whole Slide Image Analysis in Digital Pathology
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；NVIDIA Corporation（英伟达公司）；Georgia State University（佐治亚州立大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Weill Cornell Medicine（威尔康奈尔医学院）
本文提出基于 ADIOS2 的数字病理 WSI 数据管理流水线，通过并行化补丁级 I/O 缩短检索时间，在 CPU 场景获两倍加速，并在 GPU 场景达到与 NVIDIA Magnum IO GDS 相当的性能。
Objective: We develop a computer-aided diagnosis (CAD) system using deep learning approaches for lesion detection and classification on whole-slide images (WSIs) with breast cancer. The deep features being distinguishing in classification from the convolutional neural networks (CNN) are demonstrated in this study to provide comprehensive interpretability for the proposed CAD system using pathological knowledge. Methods: In the experiment, a total of 186 slides of WSIs were collected and classified into three categories: Non-Carcinoma, Ductal Carcinoma in Situ (DCIS), and Invasive Ductal Carcin…
Cancer is a complex disease that provides various types of information depending on the scale of observation. While most tumor diagnostics are performed by observing histopathological slides, radiology images should yield additional knowledge towards the efficacy of cancer diagnostics. This work investigates a deep learning method combining whole slide images and magnetic resonance images to classify tumors. In particular, our solution comprises a powerful, generic and modular architecture for whole slide image classification. Experiments are prospectively conducted on the 2020 Computational P…
Feature vectors provided by pre-trained deep artificial neural networks have become a dominant source for image representation in recent literature. Their contribution to the performance of image analysis can be improved through finetuning. As an ultimate solution, one might even train a deep network from scratch with the domain-relevant images, a highly desirable option which is generally impeded in pathology by lack of labeled images and the computational expense. In this study, we propose a new network, namely KimiaNet, that employs the topology of the DenseNet with four dense blocks, fine-…
Virchow: A Million-Slide Digital Pathology Foundation Model
Paige；Microsoft Research（微软研究院）；NSW Health Pathology（新南威尔士州卫生病理学）；St George Hospital（圣乔治医院）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Rochester（罗切斯特大学）
NEC Laboratories America（NEC美国实验室）；NEC Corporation（NEC公司）；Hokkaido University Hospital（北海道大学医院）
We propose a method to accurately obtain the ratio of tumor cells over an entire histological slide. We use deep fully convolutional neural network models trained to detect and classify cells on images of H&E-stained tissue sections. Pathologists' labels consisting of exhaustive nuclei locations and tumor regions were used to trained the model in a supervised fashion. We show that combining two models, each working at a different magnification allows the system to capture both cell-level details and surrounding context to enable successful detection and classification of cells as either tumor-…
Histopathology digital scans are large-size images that contain valuable information at the pixel level. Content-based comparison of these images is a challenging task. This study proposes a content-based similarity measure for high-resolution gigapixel histopathology images. The proposed similarity measure is an expansion of cosine vector similarity to a matrix. Each image is divided into same-size patches with a meaningful amount of information (i.e., contained enough tissue). The similarity is measured by the extraction of patch-level deep embeddings of the last pooling layer of a pre-train…
University of Stavanger（斯塔万格大学）；Stavanger University Hospital（斯塔万格大学医院）
Northern Europe has the second highest mortality rate of melanoma globally. In 2020, the mortality rate of melanoma rose to 1.9 per 100 000 habitants. Melanoma prognosis is based on a pathologist's subjective visual analysis of the patient's tumor. This methodology is heavily time-consuming, and the prognosis variability among experts is notable, drastically jeopardizing its reproducibility. Thus, the need for faster and more reproducible methods arises. Machine learning has paved its way into digital pathology, but so far, most contributions are on localization, segmentation, and diagnostics,…
Stanford School of Medicine（斯坦福医学院）；Center for Artificial Intelligence in Medicine & Imaging（医学与人工智能中心）；Department of Urology（泌尿外科）；Institute for Pathology and Cytology, Schüttorf（许特霍夫病理与细胞研究所）；Department of Radiology（放射科）；Department of Biomedical Data Science（生物医学数据科学系）
Different convolutional neural network (CNN) models have been tested for their application in histological image analyses. However, these models are prone to overfitting due to their large parameter capacity, requiring more data or valuable computational resources for model training. Given these limitations, we introduced a novel architecture (termed PlexusNet). We utilized 310 Hematoxylin and Eosin stained (H&E) annotated histological images of prostate cancer cases from TCGA-PRAD and Stanford University and 398 H&E whole slides images from the Camelyon 2016 challenge. PlexusNet-architecture …
