Human-machine Interactive Tissue Prototype Learning for Label-efficient Histopathology Image Segmentation
Tsinghua Shenzhen International Graduate School（清华深圳国际研究生院）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）；University of California, Los Angeles（加州大学洛杉矶分校）；The Chinese University of Hong Kong（香港中文大学）
Recently, deep neural networks have greatly advanced histopathology image segmentation but usually require abundant annotated data. However, due to the gigapixel scale of whole slide images and pathologists' heavy daily workload, obtaining pixel-level labels for supervised learning in clinical practice is often infeasible. Alternatively, weakly-supervised segmentation methods have been explored with less laborious image-level labels, but their performance is unsatisfactory due to the lack of dense supervision. Inspired by the recent success of self-supervised learning methods, we present a label-efficient tissue prototype dictionary building pipeline and propose to use the obtained prototypes to guide histopathology image segmentation. Particularly, taking advantage of self-supervised contrastive learning, an encoder is trained to project the unlabeled histopathology image patches into a discriminative embedding space where these patches are clustered to identify the tissue prototypes by efficient pathologists' visual examination. Then, the encoder is used to map the images into the embedding space and generate pixel-level pseudo tissue masks by querying the tissue prototype dictionary. Finally, the pseudo masks are used to train a segmentation network with dense supervision for better performance. Experiments on two public datasets demonstrate that our human-machine interactive tissue prototype learning method can achieve comparable segmentation performance as the fully-supervised baselines with less annotation burden and outperform other weakly-supervised methods. Codes will be available upon publication.
Guangdong Cardiovascular Institute（广东省心血管病研究所）；School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）
Tissue-level semantic segmentation is a vital step in computational pathology. Fully-supervised models have already achieved outstanding performance with dense pixel-level annotations. However, drawing such labels on the giga-pixel whole slide images is extremely expensive and time-consuming. In this paper, we use only patch-level classification labels to achieve tissue semantic segmentation on histopathology images, finally reducing the annotation efforts. We proposed a two-step model including a classification and a segmentation phases. In the classification phase, we proposed a CAM-based mo…
Can We Go Beyond Visual Features? Neural Tissue Relation Modeling for Relational Graph Analysis in Non-Melanoma Skin Histology
Mohamed bin Zayed University of AI（Mohamed bin Zayed人工智能大学）；School of Information and Data Sciences（信息与数据科学学院）；Vellore Institute of Technology（维洛雷理工学院）；Loughborough University（洛桑大学）；MD Anderson Cancer Center, The University of Texas（MD安德森癌症中心，德克萨斯大学）
Radboud University Medical Center（拉德堡德大学医学中心）；Institute of Experimental Immunology, University of Zurich（苏黎世大学实验免疫学研究所）；Center for Medical Image Science and Visualization, Linköping University（林雪平大学医学影像科学与可视化中心）
We propose HookNet, a semantic segmentation model for histopathology whole-slide images, which combines context and details via multiple branches of encoder-decoder convolutional neural networks. Concentricpatches at multiple resolutions with different fields of view are used to feed different branches of HookNet, and intermediate representations are combined via a hooking mechanism. We describe a framework to design and train HookNet for achieving high-resolution semantic segmentation and introduce constraints to guarantee pixel-wise alignment in feature maps during hooking. We show the advan…
University of Washington（华盛顿大学）；Seattle Children’s Hospital（西雅图儿童医院）；University of Vermont（佛蒙特大学）；Pathology Associates
In this study, we propose the Ductal Instance-Oriented Pipeline (DIOP) that contains a duct-level instance segmentation model, a tissue-level semantic segmentation model, and three-levels of features for diagnostic classification. Based on recent advancements in instance segmentation and the Mask R-CNN model, our duct-level segmenter tries to identify each ductal individual inside a microscopic image; then, it extracts tissue-level information from the identified ductal instances. Leveraging three levels of information obtained from these ductal instances and also the histopathology image, the…
Radboud University Medical Center（拉德堡德大学医学中心）；Ospedale Cannizzaro（坎尼扎罗医院）；Leids Universitair Medisch Centrum（莱顿大学医学中心）；University of Bayreuth（拜罗伊特大学）；Linköping University（林雪平大学）
Artificial Intelligence (AI) can potentially support histopathologists in the diagnosis of a broad spectrum of cancer types. In colorectal cancer (CRC), AI can alleviate the laborious task of characterization and reporting on resected biopsies, including polyps, the numbers of which are increasing as a result of CRC population screening programs, ongoing in many countries all around the globe. Here, we present an approach to address two major challenges in automated assessment of CRC histopathology whole-slide images. First, we present an AI-based method to segment multiple tissue compartments…
University Hospital Essen (AöR)（埃森大学医院）；Cancer Research Center Cologne Essen (CCCE)（科隆-埃森癌症研究中心）；West German Cancer Center Essen（埃森西德癌症中心）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；University of Duisburg-Essen（杜伊斯堡-埃森大学）；Hannover Medical School（汉诺威医学院）
The segmentation of histopathological whole slide images into tumourous and non-tumourous types of tissue is a challenging task that requires the consideration of both local and global spatial contexts to classify tumourous regions precisely. The identification of subtypes of tumour tissue complicates the issue as the sharpness of separation decreases and the pathologist's reasoning is even more guided by spatial context. However, the identification of detailed types of tissue is crucial for providing personalized cancer therapies. Due to the high resolution of whole slide images, existing sem…
Semantic Segmentation Based Quality Control of Histopathology Whole Slide Images
Indian Institute of Technology - Bombay（印度理工学院-博伊斯班加分校）；Department of Electrical Engineering（电气工程系）；Koita Centre for Digital Health（数字健康中心）；Indian Council of Medical Research（印度医学研究委员会）；National Institute for Research in Digital Health and Data Science（数字健康与数据科学国家研究所）；Tata Memorial Centre – ACTREC（塔塔纪念中心–ACTREC）；HBNI
University Medicine Göttingen, Department of General, Visceral and Pediatric Surgery（乌尔姆医学院，普通外科、 visceral 外科和儿童外科部门）；Georg-August-University Göttingen, Institute of Computer Science（哥廷根大学，计算机科学学院）；CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen（下萨克森AI与医学因果方法研究中心，哥廷根）；Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen（多尺度生物成像集群（MBExC），哥廷根大学）
Tissue Image Analytics Centre, Department of Computer Science, University of Warwick（华威大学计算机科学系组织图像分析中心）
From the simple measurement of tissue attributes in pathology workflow to designing an explainable diagnostic/prognostic AI tool, access to accurate semantic segmentation of tissue regions in histology images is a prerequisite. However, delineating different tissue regions manually is a laborious, time-consuming and costly task that requires expert knowledge. On the other hand, the state-of-the-art automatic deep learning models for semantic segmentation require lots of annotated training data and there are only a limited number of tissue region annotated images publicly available. To obviate …
Guangdong Provincial People’s Hospital（广东省人民医院）；Guangdong Academy of Medical Sciences（广东省医学科学院）；Guilin University of Electronic Technology（桂林电子科技大学）；Guangdong Provicial People’s Hospital Ganzhou Hospital（广东省人民医院赣州医院）
Lung cancer is the leading cause of cancer death worldwide, and adenocarcinoma (LUAD) is the most common subtype. Exploiting the potential value of the histopathology images can promote precision medicine in oncology. Tissue segmentation is the basic upstream task of histopathology image analysis. Existing deep learning models have achieved superior segmentation performance but require sufficient pixel-level annotations, which is time-consuming and expensive. To enrich the label resources of LUAD and to alleviate the annotation efforts, we organize this challenge WSSS4LUAD to call for the outs…
Tissue Aware Nuclei Detection and Classification Model for Histopathology Images
TIA Centre, Department of Computer Science, University of Warwick, UK（沃里克大学计算机科学系TIA中心，英国）；GSK, Artificial Intelligence and Machine Learning, London, UK（葛兰素史克人工智能与机器学习，英国）
Rutgers University（罗格斯大学）；Rutgers Cancer Institute of New Jersey（新泽西罗格斯癌症研究所）
Automatic and accurate Gleason grading of histopathology tissue slides is crucial for prostate cancer diagnosis, treatment, and prognosis. Usually, histopathology tissue slides from different institutions show heterogeneous appearances because of different tissue preparation and staining procedures, thus the predictable model learned from one domain may not be applicable to a new domain directly. Here we propose to adopt unsupervised domain adaptation to transfer the discriminative knowledge obtained from the source domain to the target domain without requiring labeling of images at the target…
CellPilot: A unified approach to automatic and interactive segmentation in histopathology
School of Computation and Information Technology, Technical University of Munich（慕尼黑工业大学计算与信息技术学院）；Helmholtz Zentrum München - German Research Center for Environmental Health（亥姆霍兹慕尼黑中心——德国环境健康研究中心）；School of Biomedical Engineering and Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）
Histopathology tissue analysis is considered the gold standard in cancer diagnosis and prognosis. Given the large size of these images and the increase in the number of potential cancer cases, an automated solution as an aid to histopathologists is highly desirable. In the recent past, deep learning-based techniques have provided state of the art results in a wide variety of image analysis tasks, including analysis of digitized slides. However, the size of images and variability in histopathology tasks makes it a challenge to develop an integrated framework for histopathology image analysis. W…
Stony Brook University（石溪大学）；University of Brasília（巴西利亚大学）；Emory University（埃默里大学）
An accurate steatosis quantification with pathology tissue samples is of high clinical importance. However, such pathology measurement is manually made in most clinical practices, subject to severe reader variability due to large sampling bias and poor reproducibility. Although some computerized automated methods are developed to quantify the steatosis regions, they present limited analysis capacity for high resolution whole-slide microscopy images and accurate overlapped steatosis division. In this paper, we propose a method that extracts an individual whole tissue piece at high resolution wi…
ACS-SegNet: An Attention-Based CNN-SegFormer Segmentation Network for Tissue Segmentation in Histopathology
Research Center for Medical Image Analysis（医学图像分析研究中心）；Artificial Intelligence, Department of Medicine, Faculty of Medicine（人工智能，医学系，医学学院）；Dentistry, Danube Private University Department of Pathophysiology（牙科，丹ube私立大学病理学系）；Allergy Research, Center of Pathophysiology, Infectiology（过敏研究，病理学中心，感染学）；Immunology, Medical University of Vienna Comprehensive Center for AI in Medicine (CAIM), Medical University of Vienna（免疫学，维也纳医学大学人工智能医学中心（CAIM），维也纳医学大学）
Stony Brook University（石溪大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Stony Brook University Hospital（石溪大学医院）
Histopathology images are crucial to the study of complex diseases such as cancer. The histologic characteristics of nuclei play a key role in disease diagnosis, prognosis and analysis. In this work, we propose a sparse Convolutional Autoencoder (CAE) for fully unsupervised, simultaneous nucleus detection and feature extraction in histopathology tissue images. Our CAE detects and encodes nuclei in image patches in tissue images into sparse feature maps that encode both the location and appearance of nuclei. Our CAE is the first unsupervised detection network for computer vision applications. T…
Cell detection is a fundamental task in computational pathology that can be used for extracting high-level medical information from whole-slide images. For accurate cell detection, pathologists often zoom out to understand the tissue-level structures and zoom in to classify cells based on their morphology and the surrounding context. However, there is a lack of efforts to reflect such behaviors by pathologists in the cell detection models, mainly due to the lack of datasets containing both cell and tissue annotations with overlapping regions. To overcome this limitation, we propose and publicl…
Vector Institute（向量研究所）；University of Waterloo（滑铁卢大学）
Whole slide imaging (WSI) refers to the digitization of a tissue specimen which enables pathologists to explore high-resolution images on a monitor rather than through a microscope. The formation of tissue folds occur during tissue processing. Their presence may not only cause out-of-focus digitization but can also negatively affect the diagnosis in some cases. In this paper, we have compared five pre-trained convolutional neural networks (CNNs) of different depths as feature extractors to characterize tissue folds. We have also explored common classifiers to discriminate folded tissue against…
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）；Vector Institute（矢量研究院）
Recent advances in digital pathology have led to the need for Histopathology Image Retrieval (HIR) systems that search through databases of biopsy images to find similar cases to a given query image. These HIR systems allow pathologists to effortlessly and efficiently access thousands of previously diagnosed cases in order to exploit the knowledge in the corresponding pathology reports. Since HIR systems may have to deal with millions of gigapixel images, the extraction of compact and expressive image features must be available to allow for efficient and accurate retrieval. In this paper, we p…
Northeastern University（东北大学）；Cancer Hospital of China Medical University（中国医科大学附属肿瘤医院）；Liaoning Cancer Hospital and Institute（辽宁省肿瘤医院）
In recent years, colorectal cancer has become one of the most significant diseases that endanger human health. Deep learning methods are increasingly important for the classification of colorectal histopathology images. However, existing approaches focus more on end-to-end automatic classification using computers rather than human-computer interaction. In this paper, we propose an IL-MCAM framework. It is based on attention mechanisms and interactive learning. The proposed IL-MCAM framework includes two stages: automatic learning (AL) and interactivity learning (IL). In the AL stage, a multi-c…
Automated classification of histopathological whole-slide images (WSI) of breast tissue requires analysis at very high resolutions with a large contextual area. In this paper, we present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, ductal carcinoma in situ (DCIS), and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution patches to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stac…
Histopathology images; microscopy images of stained tissue biopsies contain fundamental prognostic information that forms the foundation of pathological analysis and diagnostic medicine. However, diagnostics from histopathology images generally rely on a visual cognitive assessment of tissue slides which implies an inherent element of interpretation and hence subjectivity. Access to digitized histopathology images enabled the development of computational systems aiming at reducing manual intervention and automating parts of pathologists' workflow. Specifically, applications of deep learning to…
University of Warwick（华威大学）；Tissue Image Analytics (TIA) Centre, University of Warwick（华威大学组织图像分析中心）
The problem of recognizing various types of tissues present in multi-gigapixel histology images is an important fundamental pre-requisite for downstream analysis of the tumor microenvironment in a bottom-up analysis paradigm for computational pathology. In this paper, we propose a deep dictionary learning approach to solve the problem of tissue phenotyping in histology images. We propose deep Multi-Resolution Dictionary Learning (deepMRDL) in order to benefit from deep texture descriptors at multiple different spatial resolutions. We show the efficacy of the proposed approach through extensive…
Radboud University Medical Center（拉德堡德大学医学中心）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；National Cancer Institute（美国国家癌症研究所）；The Harker School（哈克学校）；Mayo Clinic（梅奥诊所）
Diagnosis of breast carcinomas has so far been limited to the morphological interpretation of epithelial cells and the assessment of epithelial tissue architecture. Consequently, most of the automated systems have focused on characterizing the epithelial regions of the breast to detect cancer. In this paper, we propose a system for classification of hematoxylin and eosin (H&E) stained breast specimens based on convolutional neural networks that primarily targets the assessment of tumor-associated stroma to diagnose breast cancer patients. We evaluate the performance of our proposed system usin…
University of British Columbia（不列颠哥伦比亚大学）；Eindhoven University of Technology（埃因霍温理工大学）；Massachusetts General Hospital（马萨诸塞州综合医院）；Harvard Medical School（哈佛医学院）；BC Cancer Agency（不列颠哥伦比亚癌症机构）
Visual microscopic study of diseased tissue by pathologists has been the cornerstone for cancer diagnosis and prognostication for more than a century. Recently, deep learning methods have made significant advances in the analysis and classification of tissue images. However, there has been limited work on the utility of such models in generating histopathology images. These synthetic images have several applications in pathology including utilities in education, proficiency testing, privacy, and data sharing. Recently, diffusion probabilistic models were introduced to generate high quality ima…
The diagnosis of cancer is mainly performed by visual analysis of the pathologists, through examining the morphology of the tissue slices and the spatial arrangement of the cells. If the microscopic image of a specimen is not stained, it will look colorless and textured. Therefore, chemical staining is required to create contrast and help identify specific tissue components. During tissue preparation due to differences in chemicals, scanners, cutting thicknesses, and laboratory protocols, similar tissues are usually varied significantly in appearance. This diversity in staining, in addition to…
There is a strong need for automated systems to improve diagnostic quality and reduce the analysis time in histopathology image processing. Automated detection and classification of pathological tissue characteristics with computer-aided diagnostic systems are a critical step in the early diagnosis and treatment of diseases. Once a pathology image is scanned by a microscope and loaded onto a computer, it can be used for automated detection and classification of diseases. In this study, the DenseNet-161 and ResNet-50 pre-trained CNN models have been used to classify digital histopathology patch…
Learning from the Right Patches: A Two-Stage Wavelet-Driven Masked Autoencoder for Histopathology Representation Learning
PLRI Medical Informatics Institute（PLRI医学信息学研究所）；CAIMed Reserch Center（CAIMed研究中心）；Hannover Medical School（汉诺威医学院）；Computer Science Institute（计算机科学研究所）；Leibniz University Hannover（莱比锡大学汉诺威分校）；Sanford Burnham Prebys Medical Discovery Institute（桑福医疗发现研究所）；Rady Children’s Institute for Genomic Medicine（拉迪儿童基因医学研究所）；University of California San Diego（加州大学圣地亚哥分校）
GRAPHITE: Graph-Based Interpretable Tissue Examination for Enhanced Explainability in Breast Cancer Histopathology
School of Computer Science and Engineering（计算机科学与工程学院）；University of New South Wales（新南威尔士大学）；St. George and Sutherland Clinical School（圣乔治与萨瑟兰临床学院）；Department of Anatomical Pathology（解剖病理学部）；NSW Health Pathology（新南威尔士州卫生病理学）；Cancer Care Centre（癌症护理中心）
Automated grading of prostate cancer histopathology images is a challenging task, with one key challenge being the scarcity of annotations down to the level of regions of interest (strong labels), as typically the prostate cancer Gleason score is known only for entire tissue slides (weak labels). In this study, we focus on automated Gleason score assignment of prostate cancer whole-slide images on the basis of a large weakly-labeled dataset and a smaller strongly-labeled one. We efficiently leverage information from both label sources by jointly training a classifier on the two datasets and by…
Staining normalization in histopathology: Method benchmarking using multicenter dataset
University of Turku, Institute of Biomedicine（图尔库大学生物医学研究所）；Department of Pathology, Hospital Nova of Central Finland（中央芬兰医院病理科部）；University of Eastern Finland, Institute of Biomedicine（东部芬兰大学生物医学研究所）；Tampere University, Faculty of Medicine and Health Technology（塔皮奥拉大学医学与健康技术学院）；InFlames Research Flagship, University of Turku（InFlames研究旗舰，图尔库大学）
Preparing and scanning histopathology slides consists of several steps, each with a multitude of parameters. The parameters can vary between pathology labs and within the same lab over time, resulting in significant variability of the tissue appearance that hampers the generalization of automatic image analysis methods. Typically, this is addressed with ad-hoc approaches such as staining normalization that aim to reduce the appearance variability. In this paper, we propose a systematic solution based on domain-adversarial neural networks. We hypothesize that removing the domain information fro…
ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology Segmentation
Ho Chi Minh City University of Technology（胡志明市技术大学）；University of Technology Sydney（悉尼技术大学）；University of Houston（休斯顿大学）；University of Information Technology（信息科技大学）；College of Medical Science and Technology, Taipei Medical University（台北医学院医学科技学院）；Department of Computer Science, Emory University（埃默里大学计算机科学系）；Montefiore Medical Center, Albert Einstein College of Medicine（蒙特福伊医疗中心，阿尔伯特·爱因斯坦医学院）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）
Weakly Supervised Spatial Grounding for Discriminative Attention-Based Ultrasound-Histopathology Alignment in Prostate Cancer Grading
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Ordensklinikum Linz（林茨教团医院）；University of Alberta（阿尔伯塔大学）；Exact Imaging（Exact Imaging公司）
ProBAG: Prototype-Guided Boundary-Aware Graph Diffusion for Weakly Supervised Histopathology Segmentation
AI VIETNAM Lab（AI越南实验室）；Washington University School of Medicine（华盛顿大学医学院）；Jeonbuk National University（全北国立大学）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
Feature-interactive Siamese graph encoder-based image analysis to predict STAS from histopathology images in lung cancer
Hunan University（湖南大学）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）；Central South University（中南大学）；Hunan Clinical Medical Research Center for Cancer Pathogenic Genes Testing and Diagnosis（湖南省肿瘤致病基因检测与诊断临床医学研究中心）；Shanghai Cancer Institute（上海市肿瘤研究所）；Renji Hospital, School of Medicine, Shanghai Jiaotong University（上海交通大学医学院附属仁济医院）；Shanghai Jiaotong University（上海交通大学）
Due to the recent advancements in machine vision, digital pathology has gained significant attention. Histopathology images are distinctly rich in visual information. The tissue glass slide images are utilized for disease diagnosis. Researchers study many methods to process histopathology images and facilitate fast and reliable diagnosis; therefore, the availability of high-quality slides becomes paramount. The quality of the images can be negatively affected when the glass slides are ink-marked by pathologists to delineate regions of interest. As an example, in one of the largest public histo…
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）
Digital pathology is one of the most significant developments in modern medicine. Pathological examinations are the gold standard of medical protocols and play a fundamental role in diagnosis. Recently, with the advent of digital scanners, tissue histopathology slides can now be digitized and stored as digital images. As a result, digitized histopathological tissues can be used in computer-aided image analysis programs and machine learning techniques. Detection and segmentation of nuclei are some of the essential steps in the diagnosis of cancers. Recently, deep learning has been used for nucl…
Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization
School of Information and Communication Technology, Griffith University, QLD, Australia（信息与通信技术学院，格里菲斯大学，昆士兰，澳大利亚）；School of MDP -- Clinical Medicine, Griffith University, QLD, Australia（MDP临床医学学院，格里菲斯大学，昆士兰，澳大利亚）
New York University（纽约大学）；NYU Grossman School of Medicine（纽约大学格罗斯曼医学院）
The current study of cell architecture of inflammation in histopathology images commonly performed for diagnosis and research purposes excludes a lot of information available on the biopsy slide. In autoimmune diseases, major outstanding research questions remain regarding which cell types participate in inflammation at the tissue level, and how they interact with each other. While these questions can be partially answered using traditional methods, artificial intelligence approaches for segmentation and classification provide a much more efficient method to understand the architecture of infl…
