In this study, we propose the Ductal Instance-Oriented Pipeline (DIOP) that contains a duct-level instance segmentation model, a tissue-level semantic segmentation model, and three-levels of features for diagnostic classification. Based on recent advancements in instance segmentation and the Mask R-CNN model, our duct-level segmenter tries to identify each ductal individual inside a microscopic image; then, it extracts tissue-level information from the identified ductal instances. Leveraging three levels of information obtained from these ductal instances and also the histopathology image, the proposed DIOP outperforms previous approaches (both feature-based and CNN-based) in all diagnostic tasks; for the four-way classification task, the DIOP achieves comparable performance to general pathologists in this unique dataset. The proposed DIOP only takes a few seconds to run in the inference time, which could be used interactively on most modern computers. More clinical explorations are needed to study the robustness and generalizability of this system in the future.
Radboud University Medical Center（拉德堡德大学医学中心）；Institute of Experimental Immunology, University of Zurich（苏黎世大学实验免疫学研究所）；Center for Medical Image Science and Visualization, Linköping University（林雪平大学医学影像科学与可视化中心）
We propose HookNet, a semantic segmentation model for histopathology whole-slide images, which combines context and details via multiple branches of encoder-decoder convolutional neural networks. Concentricpatches at multiple resolutions with different fields of view are used to feed different branches of HookNet, and intermediate representations are combined via a hooking mechanism. We describe a framework to design and train HookNet for achieving high-resolution semantic segmentation and introduce constraints to guarantee pixel-wise alignment in feature maps during hooking. We show the advan…
Jalpaiguri Government Engineering College（贾尔派古里政府工程学院）
Invasive ductal carcinoma (IDC), which is also sometimes known as the infiltrating ductal carcinoma, is the most regular form of breast cancer. It accounts for about 80% of all breast cancers. According to the American Cancer Society, more than 180,000 women in the United States are diagnosed with invasive breast cancer each year. The survival rate associated with this form of cancer is about 77% to 93% depending on the stage at which they are being diagnosed. The invasiveness and the frequency of the occurrence of these disease makes it one of the difficult cancers to be diagnosed. Our propos…
Duke University（杜克大学）；Duke University Comprehensive Cancer Center（杜克大学综合癌症中心）
Purpose: To determine whether deep learning-based algorithms applied to breast MR images can aid in the prediction of occult invasive disease following the di- agnosis of ductal carcinoma in situ (DCIS) by core needle biopsy. Material and Methods: In this institutional review board-approved study, we analyzed dynamic contrast-enhanced fat-saturated T1-weighted MRI sequences of 131 patients at our institution with a core needle biopsy-confirmed diagnosis of DCIS. The patients had no preoperative therapy before breast MRI and no prior history of breast cancer. We explored two different deep lear…
Eindhoven University of Technology（埃因霍温理工大学）；University Medical Center Utrecht（乌得勒支大学医学中心）；University Utrecht（乌得勒支大学）；Harvard Medical School（哈佛医学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）
Ductal carcinoma in situ (DCIS) is a non-invasive breast cancer that can progress into invasive ductal carcinoma (IDC). Studies suggest DCIS is often overtreated since a considerable part of DCIS lesions may never progress into IDC. Lower grade lesions have a lower progression speed and risk, possibly allowing treatment de-escalation. However, studies show significant inter-observer variation in DCIS grading. Automated image analysis may provide an objective solution to address high subjectivity of DCIS grading by pathologists. In this study, we developed a deep learning-based DCIS grading sys…
We propose a virtual staining methodology based on Generative Adversarial Networks to map histopathology images of breast cancer tissue from H&E stain to PHH3 and vice versa. We use the resulting synthetic images to build Convolutional Neural Networks (CNN) for automatic detection of mitotic figures, a strong prognostic biomarker used in routine breast cancer diagnosis and grading. We propose several scenarios, in which CNN trained with synthetically generated histopathology images perform on par with or even better than the same baseline model trained with real images. We discuss the potentia…
Automated classification of histopathological whole-slide images (WSI) of breast tissue requires analysis at very high resolutions with a large contextual area. In this paper, we present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, ductal carcinoma in situ (DCIS), and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution patches to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stac…
Top Data Science Ltd.（顶级数据科学有限公司）；University of Helsinki（赫尔辛基大学）；Skåne University Hospital（斯科讷大学医院）；Helsinki University Hospital（赫尔辛基大学医院）
Deep neural networks have introduced significant advancements in the field of machine learning-based analysis of digital pathology images including prostate tissue images. With the help of transfer learning, classification and segmentation performance of neural network models have been further increased. However, due to the absence of large, extensively annotated, publicly available prostate histopathology datasets, several previous studies employ datasets from well-studied computer vision tasks such as ImageNet dataset. In this work, we propose a transfer learning scheme from breast histopath…
Radboud University Medical Center（拉德堡德大学医学中心）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；National Cancer Institute（美国国家癌症研究所）；The Harker School（哈克学校）；Mayo Clinic（梅奥诊所）
Diagnosis of breast carcinomas has so far been limited to the morphological interpretation of epithelial cells and the assessment of epithelial tissue architecture. Consequently, most of the automated systems have focused on characterizing the epithelial regions of the breast to detect cancer. In this paper, we propose a system for classification of hematoxylin and eosin (H&E) stained breast specimens based on convolutional neural networks that primarily targets the assessment of tumor-associated stroma to diagnose breast cancer patients. We evaluate the performance of our proposed system usin…
A Chain-of-thought Reasoning Breast Ultrasound Dataset Covering All Histopathology Categories
Peking University（北京大学）；Peking Union Medical College Hospital（北京大学人民医院）；Peking University Cancer Hospital & Institute（北京大学肿瘤医院）；National Cancer Center / National Clinical Research Center for Cancer / Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院国家癌症中心/国家癌症临床科研中心/癌症医院）；Shenzhen Maternity & Child Health Care Hospital（深圳妇幼保健医院）；Xi’an Jiaotong University（西安交通大学）；Yizhun Medical AI Co., Ltd（亿众医疗人工智能有限公司）
We present a unified framework to predict tumor proliferation scores from breast histopathology whole slide images. Our system offers a fully automated solution to predicting both a molecular data-based, and a mitosis counting-based tumor proliferation score. The framework integrates three modules, each fine-tuned to maximize the overall performance: An image processing component for handling whole slide images, a deep learning based mitosis detection network, and a proliferation scores prediction module. We have achieved 0.567 quadratic weighted Cohen's kappa in mitosis counting-based score p…
University Joseph Fourier（约瑟夫·傅里叶大学）；IPAL, CNRS UMI 2955（IPAL，法国国家科学研究中心UMI 2955）；CoSMo Software（CoSMo软件公司）；Temasys Communications（Temasys通信公司）；University Pierre and Marie Curie（皮埃尔和玛丽·居里大学）
Accurate detection of mitosis plays a critical role in breast cancer histopathology. Manual detection and counting of mitosis is tedious and subject to considerable inter- and intra-reader variations. Multispectral imaging is a recent medical imaging technology, proven successful in increasing the segmentation accuracy in other fields. This study aims at improving the accuracy of mitosis detection by developing a specific solution using multispectral and multifocal imaging of breast cancer histopathological data. We propose to enable clinical routine-compliant quality of mitosis discrimination…
Studies estimate that there will be 266,120 new cases of invasive breast cancer and 40,920 breast cancer induced deaths in the year of 2018 alone. Despite the pervasiveness of this affliction, the current process to obtain an accurate breast cancer prognosis is tedious and time consuming, requiring a trained pathologist to manually examine histopathological images in order to identify the features that characterize various cancer severity levels. We propose MITOS-RCNN: a novel region based convolutional neural network (RCNN) geared for small object detection to accurately grade one of the thre…
Northeastern University（东北大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Cancer Hospital of China Medical University（中国医科大学附属肿瘤医院）；Chengdu University of Information Technology（成都信息工程大学）
Breast cancer is one of the most common and deadliest cancers among women. Since histopathological images contain sufficient phenotypic information, they play an indispensable role in the diagnosis and treatment of breast cancers. To improve the accuracy and objectivity of Breast Histopathological Image Analysis (BHIA), Artificial Neural Network (ANN) approaches are widely used in the segmentation and classification tasks of breast histopathological images. In this review, we present a comprehensive overview of the BHIA techniques based on ANNs. First of all, we categorize the BHIA systems int…
Breast cancer has the highest mortality among cancers in women. Computer-aided pathology to analyze microscopic histopathology images for diagnosis with an increasing number of breast cancer patients can bring the cost and delays of diagnosis down. Deep learning in histopathology has attracted attention over the last decade of achieving state-of-the-art performance in classification and localization tasks. The convolutional neural network, a deep learning framework, provides remarkable results in tissue images analysis, but lacks in providing interpretation and reasoning behind the decisions. …
University Hospital Essen (AöR)（埃森大学医院）；Cancer Research Center Cologne Essen (CCCE)（科隆-埃森癌症研究中心）；West German Cancer Center Essen（埃森西德癌症中心）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；University of Duisburg-Essen（杜伊斯堡-埃森大学）；Hannover Medical School（汉诺威医学院）
The segmentation of histopathological whole slide images into tumourous and non-tumourous types of tissue is a challenging task that requires the consideration of both local and global spatial contexts to classify tumourous regions precisely. The identification of subtypes of tumour tissue complicates the issue as the sharpness of separation decreases and the pathologist's reasoning is even more guided by spatial context. However, the identification of detailed types of tissue is crucial for providing personalized cancer therapies. Due to the high resolution of whole slide images, existing sem…
Each woman living in the United States has about 1 in 8 chance of developing invasive breast cancer. The mitotic cell count is one of the most common tests to assess the aggressiveness or grade of breast cancer. In this prognosis, histopathology images must be examined by a pathologist using high-resolution microscopes to count the cells. Unfortunately, this can be an exhaustive task with poor reproducibility, especially for non-experts. Deep learning networks have recently been adapted to medical applications which are able to automatically localize these regions of interest. However, these r…
University of Toronto（多伦多大学）；National Cancer Research Centre（国家癌症研究中心）；Institute for Oncology and Radiology（肿瘤与放射学研究所）
Accurate diagnosis of breast cancer in histopathology images is challenging due to the heterogeneity of cancer cell growth as well as of a variety of benign breast tissue proliferative lesions. In this paper, we propose a practical and self-interpretable invasive cancer diagnosis solution. With minimum annotation information, the proposed method mines contrast patterns between normal and malignant images in unsupervised manner and generates a probability map of abnormalities to verify its reasoning. Particularly, a fully convolutional autoencoder is used to learn the dominant structural patter…
Automating tumor-infiltrating lymphocyte assessment in breast cancer histopathology images using QuPath: a transparent and accessible machine learning pipeline
UiT The Arctic University of Norway（挪威北极圈大学）；University Hospital of North Norway（挪威北部大学医院）
Center for Wireless Networks & Applications (WNA), Amrita Vishwa Vidyapeetham（阿姆里塔·维什瓦·维迪亚皮塔姆大学无线网络与应用中心）；Valeo（法雷奥）；Department of Computer Science and Engineering, Amrita Vishwa Vidyapeetham（阿姆里塔·维什瓦·维迪亚皮塔姆大学计算机科学与工程系）；School of Computing and Electrical Engineering, IIT Mandi（印度理工学院曼迪分校计算与电气工程学院）
Computer assisted diagnosis in digital pathology is becoming ubiquitous as it can provide more efficient and objective healthcare diagnostics. Recent advances have shown that the convolutional Neural Network (CNN) architectures, a well-established deep learning paradigm, can be used to design a Computer Aided Diagnostic (CAD) System for breast cancer detection. However, the challenges due to stain variability and the effect of stain normalization with such deep learning frameworks are yet to be well explored. Moreover, performance analysis with arguably more efficient network models, which may…
Reduced Deep Convolutional Activation Features (R-DeCAF) in Histopathology Images to Improve the Classification Performance for Breast Cancer Diagnosis
Institute of Medical Science and Technology, Shahid Beheshti University（沙希德·贝赫什提大学医学科学与技术研究所）
Histopathology tissue analysis is considered the gold standard in cancer diagnosis and prognosis. Given the large size of these images and the increase in the number of potential cancer cases, an automated solution as an aid to histopathologists is highly desirable. In the recent past, deep learning-based techniques have provided state of the art results in a wide variety of image analysis tasks, including analysis of digitized slides. However, the size of images and variability in histopathology tasks makes it a challenge to develop an integrated framework for histopathology image analysis. W…
BCDNet: A Fast Residual Neural Network For Invasive Ductal Carcinoma Detection
Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）
Leveraging Medical Foundation Model Features in Graph Neural Network-Based Retrieval of Breast Histopathology Images
Artificial Intelligence Department, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran；Department of Computer Engineering Techniques, Mazaya University College, Nasiriyah, Iraq；Department of Medicine, Danube Private University, Krems an der Donau, Austria；Austrian Center for Medical Innovation；Research Center for Medical Image Analysis；Artificial Intelligence, Department of Medicine, Danube Private University, Krems an der Donau, Austria
GAFR-Net: A Graph Attention and Fuzzy-Rule Network for Interpretable Breast Cancer Image Classification
1 Department of IT Engineering, Mokwon University, Daejeon 35349, South Korea；2 School of Digital Arts, Jiangxi Arts \& Ceramics Technology Institute, Jingdezhen 333001, China
Karolinska Institutet（卡罗林斯卡学院）；Tampere University（坦佩雷大学）；Lund University（隆德大学）；University of Turku（图尔库大学）；Karolinska University Laboratory（卡罗林斯卡大学实验室）
Molecular phenotyping is central in cancer precision medicine, but remains costly and standard methods only provide a tumour average profile. Microscopic morphological patterns observable in histopathology sections from tumours are determined by the underlying molecular phenotype and associated with clinical factors. The relationship between morphology and molecular phenotype has a potential to be exploited for prediction of the molecular phenotype from the morphology visible in histopathology images. We report the first transcriptome-wide Expression-MOrphology (EMO) analysis in breast cancer,…
Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathology
School of Biomedical Engineering, Faulty of Medicine, Dalian University of Technology, Dalian, China（生物医学工程学院）；Faculty of Information Technology, University of Jyvaskyla, Jyvaskyla, Finland（信息科技学院）；School of Software Technology, Dalian University of Technology, Dalian, China（软件技术学院）；Department of Radiology, Guangdong Provincial People’s Hospital, Southern Medical University, Guangzhou, China（放射科）
Breast cancer is the most commonly diagnosed cancer worldwide, with over two million new cases each year. During diagnostic tumour grading, pathologists manually count the number of dividing cells (mitotic figures) in biopsy or tumour resection specimens. Since the process is subjective and time-consuming, data-driven artificial intelligence (AI) methods have been developed to automatically detect mitotic figures. However, these methods often generalise poorly, with performance reduced by variations in tissue types, staining protocols, or the scanners used to digitise whole-slide images. Domai…
Northeastern University（东北大学）；China Medical University（中国医科大学）；University of Luebeck（吕贝克大学）
Background: Breast cancer has the highest prevalence in women globally. The classification and diagnosis of breast cancer and its histopathological images have always been a hot spot of clinical concern. In Computer-Aided Diagnosis (CAD), traditional classification models mostly use a single network to extract features, which has significant limitations. On the other hand, many networks are trained and optimized on patient-level datasets, ignoring the application of lower-level data labels. Method: This paper proposes a deep ensemble model based on image-level labels for the binary classificat…
This work presents a novel self-supervised pre-training method to learn efficient representations without labels on histopathology medical images utilizing magnification factors. Other state-of-theart works mainly focus on fully supervised learning approaches that rely heavily on human annotations. However, the scarcity of labeled and unlabeled data is a long-standing challenge in histopathology. Currently, representation learning without labels remains unexplored for the histopathology domain. The proposed method, Magnification Prior Contrastive Similarity (MPCS), enables self-supervised lear…
Objective: We develop a computer-aided diagnosis (CAD) system using deep learning approaches for lesion detection and classification on whole-slide images (WSIs) with breast cancer. The deep features being distinguishing in classification from the convolutional neural networks (CNN) are demonstrated in this study to provide comprehensive interpretability for the proposed CAD system using pathological knowledge. Methods: In the experiment, a total of 186 slides of WSIs were collected and classified into three categories: Non-Carcinoma, Ductal Carcinoma in Situ (DCIS), and Invasive Ductal Carcin…
GRAPHITE: Graph-Based Interpretable Tissue Examination for Enhanced Explainability in Breast Cancer Histopathology
School of Computer Science and Engineering（计算机科学与工程学院）；University of New South Wales（新南威尔士大学）；St. George and Sutherland Clinical School（圣乔治与萨瑟兰临床学院）；Department of Anatomical Pathology（解剖病理学部）；NSW Health Pathology（新南威尔士州卫生病理学）；Cancer Care Centre（癌症护理中心）
In this paper, we develop a complete pipeline for stain normalization, segmentation, and classification of nuclei in hematoxylin and eosin (H&E) stained breast cancer histopathology images. In the first step, we use a CNN-based stain transfer technique to normalize the staining characteristics of (H&E) images. We then train a neural network to segment images of nuclei from the H&E images. Finally, we train an Information Maximizing Generative Adversarial Network (InfoGAN) to learn visual representations of different types of nuclei and classify them in an entirely unsupervised manner. The resu…
The size of nuclei in histological preparations from excised breast tumors is predictive of patient outcome (large nuclei indicate poor outcome). Pathologists take into account nuclear size when performing breast cancer grading. In addition, the mean nuclear area (MNA) has been shown to have independent prognostic value. The straightforward approach to measuring nuclear size is by performing nuclei segmentation. We hypothesize that given an image of a tumor region with known nuclei locations, the area of the individual nuclei and region statistics such as the MNA can be reliably computed direc…
University of Geneva (UNIGE)（日内瓦大学）；University of Applied Sciences Western Switzerland (HES-SO)（瑞士西部应用科学大学）
Explanations for deep neural network predictions in terms of domain-related concepts can be valuable in medical applications, where justifications are important for confidence in the decision-making. In this work, we propose a methodology to exploit continuous concept measures as Regression Concept Vectors (RCVs) in the activation space of a layer. The directional derivative of the decision function along the RCVs represents the network sensitivity to increasing values of a given concept measure. When applied to breast cancer grading, nuclei texture emerges as a relevant concept in the detecti…
Histopathology image analysis can be considered as a Multiple instance learning (MIL) problem, where the whole slide histopathology image (WSI) is regarded as a bag of instances (i.e, patches) and the task is to predict a single class label to the WSI. However, in many real-life applications such as computational pathology, discovering the key instances that trigger the bag label is of great interest because it provides reasons for the decision made by the system. In this paper, we propose a deep convolutional neural network (CNN) model that addresses the primary task of a bag classification o…
Performance of externally validated machine learning models based on histopathology images for the diagnosis, classification, prognosis, or treatment outcome prediction in female breast cancer: A systematic review
McMaster University（麦克马斯特大学）；Mayo Clinic（梅奥诊所）；William Osler Health System（威廉·奥斯勒医疗系统）
Histopathology image embedding is an active research area in computer vision. Most of the embedding models exclusively concentrate on a specific magnification level. However, a useful task in histopathology embedding is to train an embedding space regardless of the magnification level. Two main approaches for tackling this goal are domain adaptation and domain generalization, where the target magnification levels may or may not be introduced to the model in training, respectively. Although magnification adaptation is a well-studied topic in the literature, this paper, to the best of our knowle…
