While challenging, the dense segmentation of histology images is a necessary first step to assess changes in tissue architecture and cellular morphology. Although specific convolutional neural network architectures have been applied with great success to the problem, few effectively incorporate visual context information from multiple scales. With this paper, we present a systematic comparison of different architectures to assess how including multi-scale information affects segmentation performance. A publicly available breast cancer and a locally collected prostate cancer datasets are being utilised for this study. The results support our hypothesis that visual context and scale play a crucial role in histology image classification problems.
Digital pathology provides an excellent opportunity for applying fully convolutional networks (FCNs) to tasks, such as semantic segmentation of whole slide images (WSIs). However, standard FCNs face challenges with respect to multi-resolution, inherited from the pyramid arrangement of WSIs. As a result, networks specifically designed to learn and aggregate information at different levels are desired. In this paper, we propose two novel multi-resolution networks based on the popular `U-Net' architecture, which are evaluated on a benchmark dataset for binary semantic segmentation in WSIs. The pr…
Interactive Whole Slide Images for RL-based Tumour Segmentation
Université Côte d’Azur（蔚蓝海岸大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；IBV（生物信息研究所）；Politecnico di Torino（都灵理工大学）；Bichat Hospital（比沙医院）；Assistance Publique–Hôpitaux de Paris（巴黎公共医疗集团）
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Freie Universität Berlin（柏林自由大学）
Large-scale image data such as digital whole-slide histology images pose a challenging task at annotation software solutions. Today, a number of good solutions with varying scopes exist. For cell annotation, however, we find that many do not match the prerequisites for fast annotations. Especially in the field of mitosis detection, it is assumed that detection accuracy could significantly benefit from larger annotation databases that are currently however very troublesome to produce. Further, multiple independent (blind) expert labels are a big asset for such databases, yet there is currently …
Rutgers University（罗格斯大学）；Rutgers Cancer Institute of New Jersey（新泽西罗格斯癌症研究所）
Automatic and accurate Gleason grading of histopathology tissue slides is crucial for prostate cancer diagnosis, treatment, and prognosis. Usually, histopathology tissue slides from different institutions show heterogeneous appearances because of different tissue preparation and staining procedures, thus the predictable model learned from one domain may not be applicable to a new domain directly. Here we propose to adopt unsupervised domain adaptation to transfer the discriminative knowledge obtained from the source domain to the target domain without requiring labeling of images at the target…
Digital pathology enables remote access or consults and powerful image analysis algorithms. However, the slide digitization process can create artifacts such as out-of-focus (OOF). OOF is often only detected upon careful review, potentially causing rescanning and workflow delays. Although scan-time operator screening for whole-slide OOF is feasible, manual screening for OOF affecting only parts of a slide is impractical. We developed a convolutional neural network (ConvFocus) to exhaustively localize and quantify the severity of OOF regions on digitized slides. ConvFocus was developed using ou…
SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；School of Software, Hefei University of Technology（合肥工业大学软件学院）；Image Processing Center, School of Astronautics, Beihang University（北京航空航天大学宇航学院图像处理中心）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程高精尖创新中心）
University of Rochester（罗切斯特大学）；HistoWiz Inc.（HistoWiz公司）
Automated whole slide image (WSI) tagging has become a growing demand due to the increasing volume and diversity of WSIs collected nowadays in histopathology. Various methods have been studied to classify WSIs with single tags but none of them focuses on labeling WSIs with multiple tags. To this end, we propose a novel end-to-end trainable deep neural network named Patch Transformer which can effectively predict multiple slide-level tags from WSI patches based on both the correlations and the uniqueness between the tags. Specifically, the proposed method learns patch characteristics considerin…
Paired Uterine Whole-Slide Images and Pathology Reports for Multimodal Computational Pathology
Institute of Pathology, Technical University of Munich（慕尼黑工业大学病理研究所）；Computer Aided Medical Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助医疗程序（CAMP））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Human Pathology, Juntendo University Graduate School of Medicine（顺天堂大学医学研究生院人体病理学部）；The Hong Kong University of Science and Technology（香港科技大学）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
Beijing University of Post and Telecommunication（北京邮电大学）；China-Japanese Friendship Hospital（中日友好医院）；Haohandata Technology Co.（好汉数据科技公司）
In this paper, we propose a novel method for highly efficient follicular segmentation of thyroid cytopathological WSIs. Firstly, we propose a hybrid segmentation architecture, which integrates a classifier into Deeplab V3 by adding a branch. A large amount of the WSI segmentation time is saved by skipping the irrelevant areas using the classification branch. Secondly, we merge the low scale fine features into the original atrous spatial pyramid pooling (ASPP) in Deeplab V3 to accurately represent the details in cytopathological images. Thirdly, our hybrid model is trained by a criterion-orient…
IBM Zurich Research Lab（IBM苏黎世研究实验室）；EPFL（洛桑联邦理工学院）；APHP. Centre（巴黎公立医院中心）；Cochin hospital（科钦医院）；Université de Paris（巴黎大学）；ETH Zurich（苏黎世联邦理工学院）；Uppsala University（乌普萨拉大学）
Segmenting histology images into diagnostically relevant regions is imperative to support timely and reliable decisions by pathologists. To this end, computer-aided techniques have been proposed to delineate relevant regions in scanned histology slides. However, the techniques necessitate task-specific large datasets of annotated pixels, which is tedious, time-consuming, expensive, and infeasible to acquire for many histology tasks. Thus, weakly-supervised semantic segmentation techniques are proposed to utilize weak supervision that is cheaper and quicker to acquire. In this paper, we propose…
The Chinese University of Hong Kong（香港中文大学）；the Sixth Affiliated Hospital of Sun Yat-sen University（中山大学附属第六医院）；Imsight Medical Technology Co., Ltd.（艾迈医疗科技有限公司）
The whole slide histopathology images (WSIs) play a critical role in gastric cancer diagnosis. However, due to the large scale of WSIs and various sizes of the abnormal area, how to select informative regions and analyze them are quite challenging during the automatic diagnosis process. The multi-instance learning based on the most discriminative instances can be of great benefit for whole slide gastric image diagnosis. In this paper, we design a recalibrated multi-instance deep learning method (RMDL) to address this challenging problem. We first select the discriminative instances, and then u…
Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis
University of Science and Technology of China（中国科学技术大学）；the First Affiliated Hospital of USTC（中国科学技术大学第一附属医院）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程高精尖创新中心）
Zero-shot segmentation of skin tumors in whole-slide images with vision-language foundation models
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano (HUMAN-Tech), Universitat Politécnica de Valencia（人类中心技术大学研究机构（HUMAN-Tech）、西班牙巴塞罗那理工大学）；Artikode Intelligence S.L.（Artikode Intelligence公司）
Stony Brook University（石溪大学）；University of Brasília（巴西利亚大学）；Emory University（埃默里大学）
An accurate steatosis quantification with pathology tissue samples is of high clinical importance. However, such pathology measurement is manually made in most clinical practices, subject to severe reader variability due to large sampling bias and poor reproducibility. Although some computerized automated methods are developed to quantify the steatosis regions, they present limited analysis capacity for high resolution whole-slide microscopy images and accurate overlapped steatosis division. In this paper, we propose a method that extracts an individual whole tissue piece at high resolution wi…
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）
Local representation learning has been a key challenge to promote the performance of the histopathological whole slide images analysis. The previous representation learning methods followed the supervised learning paradigm. However, manual annotation for large-scale WSIs is time-consuming and labor-intensive. Hence, the self-supervised contrastive learning has recently attracted intensive attention. The present contrastive learning methods treat each sample as a single class, which suffers from class collision problems, especially in the domain of histopathology image analysis. In this paper, …
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong Polytechnic University（香港理工大学）
Multi-instance learning (MIL) is an effective paradigm for whole-slide pathological images (WSIs) classification to handle the gigapixel resolution and slide-level label. Prevailing MIL methods primarily focus on improving the feature extractor and aggregator. However, one deficiency of these methods is that the bag contextual prior may trick the model into capturing spurious correlations between bags and labels. This deficiency is a confounder that limits the performance of existing MIL methods. In this paper, we propose a novel scheme, Interventional Bag Multi-Instance Learning (IBMIL), to a…
Yildiz Technical University（耶尔德兹理工大学）；Istanbul Medipol University Hospital（伊斯坦布尔梅迪波尔大学医院）；Istanbul Technical University（伊斯坦布尔理工大学）
The cervical cancer developing from the precancerous lesions caused by the Human Papilloma Virus (HPV) has been one of the preventable cancers with the help of periodic screening. There are two types of grading conventions widely accepted among pathologists. On the other hand, inter-observer variability is an important issue for final diagnosis. In this paper, a whole-slide image grading benchmark for cervical cancer precursor lesions is introduced. The papillae of the cervical epithelium and overlapping cell problems are handled and a tissue classification method with a novel morphological fe…
Multiple instance learning (MIL) is the preferred approach for whole slide image classification. However, most MIL approaches do not exploit the interdependencies of tiles extracted from a whole slide image, which could provide valuable cues for classification. This paper presents a novel MIL approach that exploits the spatial relationship of tiles for classifying whole slide images. To do so, a sparse map is built from tiles embeddings, and is then classified by a sparse-input CNN. It obtained state-of-the-art performance over popular MIL approaches on the classification of cancer subtype inv…
Automated classification of histopathological whole-slide images (WSI) of breast tissue requires analysis at very high resolutions with a large contextual area. In this paper, we present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, ductal carcinoma in situ (DCIS), and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution patches to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stac…
Learning suitable Whole slide images (WSIs) representations for efficient retrieval systems is a non-trivial task. The WSI embeddings obtained from current methods are in Euclidean space not ideal for efficient WSI retrieval. Furthermore, most of the current methods require high GPU memory due to the simultaneous processing of multiple sets of patches. To address these challenges, we propose a novel framework for learning binary and sparse WSI representations utilizing a deep generative modelling and the Fisher Vector. We introduce new loss functions for learning sparse and binary permutation-…
Radboud University Medical Center（拉德堡德大学医学中心）；Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）
Prostate cancer (PCa) is graded by pathologists by examining the architectural pattern of cancerous epithelial tissue on hematoxylin and eosin (H&E) stained slides. Given the importance of gland morphology, automatically differentiating between glandular epithelial tissue and other tissues is an important prerequisite for the development of automated methods for detecting PCa. We propose a new method, using deep learning, for automatically segmenting epithelial tissue in digitized prostatectomy slides. We employed immunohistochemistry (IHC) to render the ground truth less subjective and more p…
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Fudan University Shanghai Cancer Center（复旦大学附属肿瘤医院）；Zhongshan Hospital, Fudan University（复旦大学中山医院）
Digital Medical Research Center（数字医学研究中心）；School of Basic Medical Science（基础医学院）；Fudan University（复旦大学）
Multiple Instance Learning (MIL) is widely used in analyzing histopathological Whole Slide Images (WSIs). However, existing MIL methods do not explicitly model the data distribution, and instead they only learn a bag-level or instance-level decision boundary discriminatively by training a classifier. In this paper, we propose DGMIL: a feature distribution guided deep MIL framework for WSI classification and positive patch localization. Instead of designing complex discriminative network architectures, we reveal that the inherent feature distribution of histopathological image data can serve as…
Tongji University（同济大学）；National University of Defense Technology（国防科技大学）；Shanghai Skin Disease Hospital（上海市皮肤病医院）
Despite remarkable efforts been made, the classification of gigapixels whole-slide image (WSI) is severely restrained from either the constrained computing resources for the whole slides, or limited utilizing of the knowledge from different scales. Moreover, most of the previous attempts lacked of the ability of uncertainty estimation. Generally, the pathologists often jointly analyze WSI from the different magnifications. If the pathologists are uncertain by using single magnification, then they will change the magnification repeatedly to discover various features of the tissues. Motivated by…
The PAR dataset: Prostate biopsy whole slide images from an underrepresented Middle Eastern population
Koya University（科亚大学）；Karolinska Institutet（卡罗林斯卡学院）；Hawler Medical University（哈勒医科大学）；PAR Private Hospital（PAR私立医院）；University of Sulaimani（苏莱曼尼亚大学）；Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）
Due to the increasing availability of whole slide scanners facilitating digitization of histopathological tissue, there is a strong demand for the development of computer based image analysis systems. In this work, the focus is on the segmentation of the glomeruli constituting a highly relevant structure in renal histopathology, which has not been investigated before in combination with CNNs. We propose two different CNN cascades for segmentation applications with sparse objects. These approaches are applied to the problem of glomerulus segmentation and compared with conventional fully-convolu…
The Hong Kong University of Science and Technology（香港科技大学）；Shanghai Jiao Tong University（上海交通大学）
Multiple instance learning (MIL) has emerged as a popular method for classifying histopathology whole slide images (WSIs). However, existing approaches typically rely on pre-trained models from large natural image datasets, such as ImageNet, to generate instance features, which can be sub-optimal due to the significant differences between natural images and histopathology images that lead to a domain shift. In this paper, we present a novel, simple yet effective method for learning domain-specific knowledge transformation from pre-trained models to histopathology images. Our approach entails u…
The College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education（教育部脑机智能技术重点实验室）
The size of nuclei in histological preparations from excised breast tumors is predictive of patient outcome (large nuclei indicate poor outcome). Pathologists take into account nuclear size when performing breast cancer grading. In addition, the mean nuclear area (MNA) has been shown to have independent prognostic value. The straightforward approach to measuring nuclear size is by performing nuclei segmentation. We hypothesize that given an image of a tumor region with known nuclei locations, the area of the individual nuclei and region statistics such as the MNA can be reliably computed direc…
The application of deep learning to pathology assumes the existence of digital whole slide images of pathology slides. However, slide digitization is bottlenecked by the high cost of precise motor stages in slide scanners that are needed for position information used for slide stitching. We propose GloFlow, a two-stage method for creating a whole slide image using optical flow-based image registration with global alignment using a computationally tractable graph-pruning approach. In the first stage, we train an optical flow predictor to predict pairwise translations between successive video fr…
GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image
Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Clinical Oncology School of Fujian Medical University（福建医科大学临床肿瘤学院）；Fujian Cancer Hospital（福建省肿瘤医院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
ETH Zurich（苏黎世联邦理工学院）；Uppsala University（乌普萨拉大学）；IBM Research Europe（IBM欧洲研究院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Multiple Instance Learning (MIL) methods have become increasingly popular for classifying giga-pixel sized Whole-Slide Images (WSIs) in digital pathology. Most MIL methods operate at a single WSI magnification, by processing all the tissue patches. Such a formulation induces high computational requirements, and constrains the contextualization of the WSI-level representation to a single scale. A few MIL methods extend to multiple scales, but are computationally more demanding. In this paper, inspired by the pathological diagnostic process, we propose ZoomMIL, a method that learns to perform mu…
The widespread adoption of whole slide imaging has increased the demand for effective and efficient gigapixel image analysis. Deep learning is at the forefront of computer vision, showcasing significant improvements over previous methodologies on visual understanding. However, whole slide images have billions of pixels and suffer from high morphological heterogeneity as well as from different types of artefacts. Collectively, these impede the conventional use of deep learning. For the clinical translation of deep learning solutions to become a reality, these challenges need to be addressed. In…
Tsinghua University（清华大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peking University（北京大学）
Multiple instance learning (MIL) is a powerful tool to solve the weakly supervised classification in whole slide image (WSI) based pathology diagnosis. However, the current MIL methods are usually based on independent and identical distribution hypothesis, thus neglect the correlation among different instances. To address this problem, we proposed a new framework, called correlated MIL, and provided a proof for convergence. Based on this framework, we devised a Transformer based MIL (TransMIL), which explored both morphological and spatial information. The proposed TransMIL can effectively dea…
TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image Analysis
School of Electrical Engineering, Korea University（韩国大学电气工程学院）；Department of Hospital Pathology, The Catholic University of Korea College of Medicine（韩国天主教大学医学院医院病理学系）
