Deep learning models that are trained on histopathological images obtained from a single lab and/or scanner give poor inference performance on images obtained from another scanner/lab with a different staining protocol. In recent years, there has been a good amount of research done for image stain normalization to address this issue. In this work, we present a novel approach for the stain normalization problem using fast neural style transfer coupled with adversarial loss. We also propose a novel stain transfer generator network based on High-Resolution Network (HRNet) which requires less training time and gives good generalization with few paired training images of reference stain and test stain. This approach has been tested on Whole Slide Images (WSIs) obtained from 8 different labs, where images from one lab were treated as a reference stain. A deep learning model was trained on this stain and the rest of the images were transferred to it using the corresponding stain transfer generator network. Experimentation suggests that this approach is able to successfully perform stain normalization with good visual quality and provides better inference performance compared to not applying stain normalization.
University of Strasbourg(斯特拉斯堡大学);University of Belgrade(贝尔格莱德大学);University of Kragujevac(克拉古耶瓦茨大学);Hannover Medical School(汉诺威医学院);University Clinic, Freiburg(弗赖堡大学诊所)
The application of supervised deep learning methods in digital pathology is limited due to their sensitivity to domain shift. Digital Pathology is an area prone to high variability due to many sources, including the common practice of evaluating several consecutive tissue sections stained with different staining protocols. Obtaining labels for each stain is very expensive and time consuming as it requires a high level of domain knowledge. In this article, we propose an unsupervised augmentation approach based on adversarial image-to-image translation, which facilitates the training of stain in…
Seoul National University(首尔国立大学);Korea University(高丽大学);Yonsei University(延世大学);KAIST(韩国科学技术院)
Performance of data-driven network for tumor classification varies with stain-style of histopathological images. This article proposes the stain-style transfer (SST) model based on conditional generative adversarial networks (GANs) which is to learn not only the certain color distribution but also the corresponding histopathological pattern. Our model considers feature-preserving loss in addition to well-known GAN loss. Consequently our model does not only transfers initial stain-styles to the desired one but also prevent the degradation of tumor classifier on transferred images. The model is …
The University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);The Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院)
Histopathological analysis is crucial in artery characterization for coronary artery disease (CAD). However, histology requires an invasive and time-consuming process. In this paper, we propose to generate virtual histology staining using Optical Coherence Tomography (OCT) images to enable real-time histological visualization. We develop a deep learning network, namely Coronary-GAN, to transfer coronary OCT images to virtual histology images. With a special consideration on the structural constraints in coronary OCT images, our method achieves better image generation performance than the conve…
While hematoxylin and eosin (H&E) is a standard staining procedure, immunohistochemistry (IHC) staining further serves as a diagnostic and prognostic method. However, acquiring special staining results requires substantial costs. Hence, we proposed a strategy for ultra-high-resolution unpaired image-to-image translation: Kernelized Instance Normalization (KIN), which preserves local information and successfully achieves seamless stain transformation with constant GPU memory usage. Given a patch, corresponding position, and a kernel, KIN computes local statistics using convolution operation. In…
Technical University Munich(慕尼黑工业大学);Helmholtz AI(亥姆霍兹人工智能研究所);Munich School of Data Science (MuDS)(慕尼黑数据科学学院);ContextVision AB(ContextVision公司);Institute for Computational Biology, HelmholtzZentrum Munich(慕尼黑亥姆霍兹中心计算生物学研究所)
In digital pathology, different staining procedures and scanners cause substantial color variations in whole-slide images (WSIs), especially across different laboratories. These color shifts result in a poor generalization of deep learning-based methods from the training domain to external pathology data. To increase test performance, stain normalization techniques are used to reduce the variance between training and test domain. Alternatively, color augmentation can be applied during training leading to a more robust model without the extra step of color normalization at test time. We propose…
Technische Universität München(慕尼黑工业大学);Johns Hopkins University(约翰斯·霍普金斯大学)
Digitized Histological diagnosis is in increasing demand. However, color variations due to various factors are imposing obstacles to the diagnosis process. The problem of stain color variations is a well-defined problem with many proposed solutions. Most of these solutions are highly dependent on a reference template slide. We propose a deep-learning solution inspired by CycleGANs that is trained end-to-end, eliminating the need for an expert to pick a representative reference slide. Our approach showed superior results quantitatively and qualitatively against the state of the art methods (10%…
University of Toronto(多伦多大学);University of Alberta(阿尔伯塔大学)
Computational histopathology image diagnosis becomes increasingly popular and important, where images are segmented or classified for disease diagnosis by computers. While pathologists do not struggle with color variations in slides, computational solutions usually suffer from this critical issue. To address the issue of color variations in histopathology images, this study proposes two stain style transfer models, SSIM-GAN and DSCSI-GAN, based on the generative adversarial networks. By cooperating structural preservation metrics and feedback of an auxiliary diagnosis net in learning, medical-…
Roche Innovation Center Munich(慕尼黑罗氏创新中心);Technische Universität München(慕尼黑工业大学)
Style transfer is a field with growing interest and use cases in deep learning. Recent work has shown Generative Adversarial Networks(GANs) can be used to create realistic images of virtually stained slide images in digital pathology with clinically validated interpretability. Digital pathology images are typically of extremely high resolution, making tilewise analysis necessary for deep learning applications. It has been shown that image generators with instance normalization can cause a tiling artifact when a large image is reconstructed from the tilewise analysis. We introduce a novel perce…
Learned Image Compression for HE-stained Histopathological Images via Stain Deconvolution
German Cancer Research Center (DKFZ)(德国癌症研究中心);German Cancer Consortium (DKTK)(德国癌症联盟);National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心);Heidelberg University(海德堡大学);Technical University of Munich(慕尼黑工业大学)
Stain Style Transfer using Transitive Adversarial Networks
College of Physics and Information Engineering, Fuzhou University(福州大学物理与信息工程学院);Fujian Key Lab of Medical Instrumentation & Pharmaceutical Technology(福建省医疗器械与制药技术重点实验室);Imperial Vision Technology(英视科技);Fujian Provincial Cancer Hospital, The Affiliated Hospital of Fujian Medical University(福建医科大学附属福建省肿瘤医院)
University of Strasbourg(斯特拉斯堡大学);University of Belgrade(贝尔格莱德大学);University of Kragujevac(克拉古耶瓦茨大学)
Domain shift in digital histopathology can occur when different stains or scanners are used, during stain translation, etc. A deep neural network trained on source data may not generalise well to data that has undergone some domain shift. An important step towards being robust to domain shift is the ability to detect and measure it. This article demonstrates that the PixelCNN and domain shift metric can be used to detect and quantify domain shift in digital histopathology, and they demonstrate a strong correlation with generalisation performance. These findings pave the way for a mechanism to …
Inception Institute of Artificial Intelligence(起源人工智能研究院);EPFL(洛桑联邦理工学院);Lausanne University Hospital(洛桑大学医院);Center of Biomedical Imaging(生物医学成像中心);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Although generative adversarial network (GAN) based style transfer is state of the art in histopathology color-stain normalization, they do not explicitly integrate structural information of tissues. We propose a self-supervised approach to incorporate semantic guidance into a GAN based stain normalization framework and preserve detailed structural information. Our method does not require manual segmentation maps which is a significant advantage over existing methods. We integrate semantic information at different layers between a pre-trained semantic network and the stain color normalization …
Advancing H&E-to-IHC Stain Translation in Breast Cancer: A Multi-Magnification and Attention-Based Approach
Fudan University(复旦大学);College of Electronic Engineering, Ocean University of China(中国海洋大学电子工程学院);Research Center for Frontier Fundamental Studies, Zhejiang Lab(之江实验室前沿基础研究中心);Institutes of Biomedical Sciences, Fudan University(复旦大学生物医学研究院)
Queen Mary University of London(伦敦玛丽女王大学);Karolinska University Hospital(卡罗林斯卡大学医院)
Histopathological cancer diagnosis is based on visual examination of stained tissue slides. Hematoxylin and eosin (H\&E) is a standard stain routinely employed worldwide. It is easy to acquire and cost effective, but cells and tissue components show low-contrast with varying tones of dark blue and pink, which makes difficult visual assessments, digital image analysis, and quantifications. These limitations can be overcome by IHC staining of target proteins of the tissue slide. IHC provides a selective, high-contrast imaging of cells and tissue components, but their use is largely limited by a …
Arizona State University(亚利桑那州立大学);Barrow Neurological Institute(巴罗神经学研究所);St. Joseph’s Hospital and Medical Center(圣约瑟夫医院和医疗中心)
Confocal laser endomicroscopy (CLE) allow on-the-fly in vivo intraoperative imaging in a discreet field of view, especially for brain tumors, rather than extracting tissue for examination ex vivo with conventional light microscopy. Fluorescein sodium-driven CLE imaging is more interactive, rapid, and portable than conventional hematoxylin and eosin (H&E)-staining. However, it has several limitations: CLE images may be contaminated with artifacts (motion, red blood cells, noise), and neuropathologists are mainly trained on colorful stained histology slides like H&E while the CLE images are gray…
University of Houston(休斯顿大学);University Hospital of Cologne(科隆大学医院)
The classification of Antibody Mediated Rejection (AMR) in kidney transplant remains challenging even for experienced nephropathologists; this is partly because histological tissue stain analysis is often characterized by low inter-observer agreement and poor reproducibility. One of the implicated causes for inter-observer disagreement is the variability of tissue stain quality between (and within) pathology labs, coupled with the gradual fading of archival sections. Variations in stain colors and intensities can make tissue evaluation difficult for pathologists, ultimately affecting their abi…
GenSelfDiff-HIS: Generative Self-Supervision Using Diffusion for Histopathological Image Segmentation
Indian Institute of Science (IISc)(印度科学学院);Indian Institute of Technology Delhi (IIT-D)(印度德里理工学院);All India Institute of Medical Sciences Delhi (AIIMS)(全印度医学科学院新德里)
ADNP-15: An Open-Source Histopathological Dataset for Neuritic Plaque Segmentation in Human Brain Whole Slide Images with Frequency Domain Image Enhancement for Stain Normalization
School of Software Engineering, Beijing University of Technology(北京工业大学软件学院);Sorbonne Université(索邦大学);Institut du Cerveau - Paris Brain Institute - ICM(巴黎脑研究所-巴黎脑研究所-ICM);Inserm(法国国家医学院);CNRS(法国国家科学研究中心);APHP(法国国家卫生与医学研究院);Hôpital de la Pitié Salpêtrière(皮蒂埃萨尔佩特里医院);DMU Neuroscience(DMU神经科学);Inria(法国国家信息与自动化技术研究院)
Score-based Diffusion Model for Unpaired Virtual Histology Staining
Department of Health Technology and Informatics, Hong Kong Polytechnic University, China(香港理工大学健康科技与信息学系);Department of Clinical Neurosciences, University of Cambridge, UK(剑桥大学临床神经科学系);School of Science and Engineering, University of Dundee, UK(邓迪大学科学与工程学院);Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK(剑桥大学应用数学与理论物理系)
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Novartis Pharmaceuticals(诺华制药);University of California, Irvine(加州大学欧文分校);Wolfram Research(沃尔夫勒姆研究公司)
Skin cancer, the most commonly found human malignancy, is primarily diagnosed visually via dermoscopic analysis, biopsy, and histopathological examination. However, unlike other types of cancer, automated image classification of skin lesions is deemed more challenging due to the irregularity and variability in the lesions' appearances. In this work, we propose an adaptation of the Neural Style Transfer (NST) as a novel image pre-processing step for skin lesion classification problems. We represent each dermoscopic image as the style image and transfer the style of the lesion onto a homogeneous…
Computational histopathology has made significant strides in the past few years, slowly getting closer to clinical adoption. One area of benefit would be the automatic generation of diagnostic reports from H\&E-stained whole slide images which would further increase the efficiency of the pathologists' routine diagnostic workflows. In this study, we compiled a dataset (PatchGastricADC22) of histopathological captions of stomach adenocarcinoma endoscopic biopsy specimens, which we extracted from diagnostic reports and paired with patches extracted from the associated whole slide images. The data…
Department of Clinical Laboratory, Tongji Hospital, Huazhong University of Science and Technology(华中科技大学同济医院检验科)
The different stain styles of cytopathological images have a negative effect on the generalization ability of automated image analysis algorithms. This article proposes a new framework that normalizes the stain style for cytopathological images through a stain removal module and a multi-stage domain adversarial style reconstruction module. We convert colorful images into grayscale images with a color-encoding mask. Using the mask, reconstructed images retain their basic color without red and blue mixing, which is important for cytopathological image interpretation. The style reconstruction mod…
Accurate segmentation of medical images is essential for diagnosis and treatment of diseases. These problems are solved by highly complex models, such as deep networks (DN), requiring a large amount of labeled data for training. Thereby, many DNs possess task- or imaging modality specific architectures with a decision-making process that is often hard to explain and interpret. Here, we propose a framework that embeds existing DNs into a low-dimensional subspace induced by the learnable explicit feature map (LEFM) layer. Compared to the existing DN, the framework adds one hyperparameter and onl…
Unpaired Multi-Domain Histopathology Virtual Staining using Dual Path Prompted Inversion
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);National innovation center for advanced medical devices(国家高性能医疗器械创新中心)
The diagnosis of cancer is mainly performed by visual analysis of the pathologists, through examining the morphology of the tissue slices and the spatial arrangement of the cells. If the microscopic image of a specimen is not stained, it will look colorless and textured. Therefore, chemical staining is required to create contrast and help identify specific tissue components. During tissue preparation due to differences in chemicals, scanners, cutting thicknesses, and laboratory protocols, similar tissues are usually varied significantly in appearance. This diversity in staining, in addition to…
Huazhong University of Science and Technology(华中科技大学);Hubei Maternal and Child Health Hospital(湖北省妇幼保健院)
Stain normalization often refers to transferring the color distribution of the source image to that of the target image and has been widely used in biomedical image analysis. The conventional stain normalization is regarded as constructing a pixel-by-pixel color mapping model, which only depends on one reference image, and can not accurately achieve the style transformation between image datasets. In principle, this style transformation can be well solved by the deep learning-based methods due to its complicated network structure, whereas, its complicated structure results in the low computati…
Histopathological characterization of colorectal polyps allows to tailor patients' management and follow up with the ultimate aim of avoiding or promptly detecting an invasive carcinoma. Colorectal polyps characterization relies on the histological analysis of tissue samples to determine the polyps malignancy and dysplasia grade. Deep neural networks achieve outstanding accuracy in medical patterns recognition, however they require large sets of annotated training images. We introduce UniToPatho, an annotated dataset of 9536 hematoxylin and eosin (H&E) stained patches extracted from 292 whole-…
Synthetic generation of three-dimensional cell models from histopathological images aims to enhance understanding of cell mutation, and progression of cancer, necessary for clinical assessment and optimal treatment. Classical reconstruction algorithms based on image registration of consecutive slides of stained tissues are prone to errors and often not suitable for the training of three-dimensional segmentation algorithms. We propose a novel framework to generate synthetic three-dimensional histological models based on a generator-discriminator pattern optimizing constrained features that cons…
Institute of Molecular and Cell Biology(分子与细胞生物学研究所);Bioinformatics Institute(生物信息学研究所);National University of Singapore(新加坡国立大学);University of Queensland(昆士兰大学);Institute for Infocomm Research(资讯通信研究所);Monash University Malaysia(莫纳什大学马来西亚分校);Tan Tock Seng Hospital(陈笃生医院);PanAsia Surgery Pte Ltd(泛亚外科私人有限公司);Mount Elizabeth Novena Hospital(伊丽莎白诺维娜医院);Surgi-TEN Specialists Pte Ltd(Surgi-TEN专科私人有限公司);Farrer Park Hospital(花拉公园医院);CNRS UMI 2955, Image & Pervasive Access Lab(法国国家科学研究中心UMI 2955,图像与普适访问实验室);Singapore Eye Research Institute(新加坡眼科研究所)
Architecture, size, and shape of glands are most important patterns used by pathologists for assessment of cancer malignancy in prostate histopathological tissue slides. Varying structures of glands along with cumbersome manual observations may result in subjective and inconsistent assessment. Cribriform gland with irregular border is an important feature in Gleason pattern 4. We propose using deep neural networks for cribriform pattern classification in prostate histopathological images. $163708$ Hematoxylin and Eosin (H\&E) stained images were extracted from histopathologic tissue slides of …
TU Darmstadt(达姆施塔特工业大学);University Hospital Cologne(科隆大学医院)
Although deep federated learning has received much attention in recent years, progress has been made mainly in the context of natural images and barely for computational pathology. However, deep federated learning is an opportunity to create datasets that reflect the data diversity of many laboratories. Further, the effort of dataset construction can be divided among many. Unfortunately, existing algorithms cannot be easily applied to computational pathology since previous work presupposes that data distributions of laboratories must be similar. This is an unlikely assumption, mainly since dif…
Nagoya Institute of Technology(名古屋工业大学);Nagoya University Hospital(名古屋大学医院);RIKEN(理化学研究所)
We propose a new method for cancer subtype classification from histopathological images, which can automatically detect tumor-specific features in a given whole slide image (WSI). The cancer subtype should be classified by referring to a WSI, i.e., a large-sized image (typically 40,000x40,000 pixels) of an entire pathological tissue slide, which consists of cancer and non-cancer portions. One difficulty arises from the high cost associated with annotating tumor regions in WSIs. Furthermore, both global and local image features must be extracted from the WSI by changing the magnifications of th…
Stony Brook University(石溪大学);Dougherty Valley High School(多尔蒂谷高中);Oak Ridge National Laboratory(橡树岭国家实验室);Stony Brook Hospital(石溪医院)
Hematoxylin and Eosin stained histopathology image analysis is essential for the diagnosis and study of complicated diseases such as cancer. Existing state-of-the-art approaches demand extensive amount of supervised training data from trained pathologists. In this work we synthesize in an unsupervised manner, large histopathology image datasets, suitable for supervised training tasks. We propose a unified pipeline that: a) generates a set of initial synthetic histopathology images with paired information about the nuclei such as segmentation masks; b) refines the initial synthetic images throu…
University of Saskatchewan(萨斯喀彻温大学);University of Tulane(杜兰大学)
Breast cancer is one of the leading causes of death across the world in women. Early diagnosis of this type of cancer is critical for treatment and patient care. Computer-aided detection (CAD) systems using convolutional neural networks (CNN) could assist in the classification of abnormalities. In this study, we proposed an ensemble deep learning-based approach for automatic binary classification of breast histology images. The proposed ensemble model adapts three pre-trained CNNs, namely VGG19, MobileNet, and DenseNet. The ensemble model is used for the feature representation and extraction s…
Massachusetts Institute of Technology(麻省理工学院);Harvard Medical School(哈佛医学院);Boston University School of Medicine(波士顿大学医学院);VA Boston Healthcare(波士顿退伍军人医疗中心)
Histopathological diagnoses of tumors in tissue biopsy after Hematoxylin and Eosin (H&E) staining is the gold standard for oncology care. H&E staining is slow and uses dyes, reagents and precious tissue samples that cannot be reused. Thousands of native nonstained RGB Whole Slide Image (RWSI) patches of prostate core tissue biopsies were registered with their H&E stained versions. Conditional Generative Adversarial Neural Networks (cGANs) that automate conversion of native nonstained RWSI to computational H&E stained images were then trained. High similarities between computational and H&E dye…
Pancreatic cancer will soon be the second leading cause of cancer-related death in Western society. Imaging techniques such as CT, MRI and ultrasound typically help providing the initial diagnosis, but histopathological assessment is still the gold standard for final confirmation of disease presence and prognosis. In recent years machine learning approaches and pathomics pipelines have shown potential in improving diagnostics and prognostics in other cancerous entities, such as breast and prostate cancer. A crucial first step in these pipelines is typically identification and segmentation of t…
South China University of Technology(华南理工大学);Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences(广东省人民医院(广东省医学科学院));Southern Medical University(南方医科大学)
Color inconsistency is an inevitable challenge in computational pathology, which generally happens because of stain intensity variations or sections scanned by different scanners. It harms the pathological image analysis methods, especially the learning-based models. A series of approaches have been proposed for stain normalization. However, most of them are lack flexibility in practice. In this paper, we formulated stain normalization as a digital re-staining process and proposed a self-supervised learning model, which is called RestainNet. Our network is regarded as a digital restainer which…
Improving Generalization Capability of Deep Learning-Based Nuclei Instance Segmentation by Non-deterministic Train Time and Deterministic Test Time Stain Normalization
Danube Private University(多瑙私立大学);Medical University of Vienna(维也纳医科大学);Austrian Center for Medical Innovation and Technology(奥地利医学创新与技术中心)
University of California, Davis(加州大学戴维斯分校);University of San Francisco Data Institute(旧金山大学数据研究所);Wicklow AI in Medicine Research Initiative(威克洛医学人工智能研究计划);University of Texas, San Antonio Health(德克萨斯大学圣安东尼奥健康分校)
MUSE is a novel slide-free imaging technique for histological examination of tissues that can serve as an alternative to traditional histology. In order to bridge the gap between MUSE and traditional histology, we aim to convert MUSE images to resemble authentic hematoxylin- and eosin-stained (H&E) images. We evaluated four models: a non-machine-learning-based color-mapping unmixing-based tool, CycleGAN, DualGAN, and GANILLA. CycleGAN and GANILLA provided visually compelling results that appropriately transferred H&E style and preserved MUSE content. Based on training an automated critic on re…
École de Technologie Superiéure, Université du Québec(魁北克大学高等技术学院);State University of Ponta Grossa (UEPG)(蓬塔格罗萨州立大学);Pontifical Catholic University of Paraná(巴拉那天主教大学);Federal University of Paraná(巴拉那联邦大学)
Data imbalance is a major problem that affects several machine learning (ML) algorithms. Such a problem is troublesome because most of the ML algorithms attempt to optimize a loss function that does not take into account the data imbalance. Accordingly, the ML algorithm simply generates a trivial model that is biased toward predicting the most frequent class in the training data. In the case of histopathologic images (HIs), both low-level and high-level data augmentation (DA) techniques still present performance issues when applied in the presence of inter-patient variability; whence the model…
Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València(瓦伦西亚理工大学生物工程研究与创新研究所);Hospital Universitario y Politécnico La Fe(拉费大学理工医院)
Recently, bladder cancer has been significantly increased in terms of incidence and mortality. Currently, two subtypes are known based on tumour growth: non-muscle invasive (NMIBC) and muscle-invasive bladder cancer (MIBC). In this work, we focus on the MIBC subtype because it is of the worst prognosis and can spread to adjacent organs. We present a self-learning framework to grade bladder cancer from histological images stained via immunohistochemical techniques. Specifically, we propose a novel Deep Convolutional Embedded Attention Clustering (DCEAC) which allows classifying histological pat…
Leveraging Pre-trained Models for FF-to-FFPE Histopathological Image Translation
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);The First Affiliated Hospital of Sun Yat-sen University(中山大学附属第一医院);Jinfeng Laboratory(金凤实验室)
Predicting EGFR Mutation in LUAD from Histopathological Whole-Slide Images Using Pretrained Foundation Model and Transfer Learning: An Indian Cohort Study
University of California, Los Angeles(加州大学洛杉矶分校);California NanoSystems Institute, University of California, Los Angeles(加州大学洛杉矶分校加州纳米系统研究所);David Geffen School of Medicine, University of California, Los Angeles(加州大学洛杉矶分校大卫·格芬医学院)
Deep learning-based virtual staining was developed to introduce image contrast to label-free tissue sections, digitally matching the histological staining, which is time-consuming, labor-intensive, and destructive to tissue. Standard virtual staining requires high autofocusing precision during the whole slide imaging of label-free tissue, which consumes a significant portion of the total imaging time and can lead to tissue photodamage. Here, we introduce a fast virtual staining framework that can stain defocused autofluorescence images of unlabeled tissue, achieving equivalent performance to v…