Histological staining is a vital step used to diagnose various diseases and has been used for more than a century to provide contrast to tissue sections, rendering the tissue constituents visible for microscopic analysis by medical experts. However, this process is time-consuming, labor-intensive, expensive and destructive to the specimen. Recently, the ability to virtually-stain unlabeled tissue sections, entirely avoiding the histochemical staining step, has been demonstrated using tissue-stain specific deep neural networks. Here, we present a new deep learning-based framework which generates virtually-stained images using label-free tissue, where different stains are merged following a micro-structure map defined by the user. This approach uses a single deep neural network that receives two different sources of information at its input: (1) autofluorescence images of the label-free tissue sample, and (2) a digital staining matrix which represents the desired microscopic map of different stains to be virtually generated at the same tissue section. This digital staining matrix is also used to virtually blend existing stains, digitally synthesizing new histological stains. We trained and blindly tested this virtual-staining network using unlabeled kidney tissue sections to generate micro-structured combinations of Hematoxylin and Eosin (H&E), Jones silver stain, and Masson's Trichrome stain. Using a single network, this approach multiplexes virtual staining of label-free tissue with multiple types of stains and paves the way for synthesizing new digital histological stains that can be created on the same tissue cross-section, which is currently not feasible with standard histochemical staining methods.
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；University of California at Davis（加州大学戴维斯分校）
The immunohistochemical (IHC) staining of the human epidermal growth factor receptor 2 (HER2) biomarker is widely practiced in breast tissue analysis, preclinical studies and diagnostic decisions, guiding cancer treatment and investigation of pathogenesis. HER2 staining demands laborious tissue treatment and chemical processing performed by a histotechnologist, which typically takes one day to prepare in a laboratory, increasing analysis time and associated costs. Here, we describe a deep learning-based virtual HER2 IHC staining method using a conditional generative adversarial network that is…
Virtual Gram staining of label-free bacteria using darkfield microscopy and deep learning
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；Texas A&M University（德克萨斯农工大学）；Texas A&M Engineering Experiment Station（德克萨斯农工大学工程实验站）；David Geffen School of Medicine（大卫·格芬医学院）
Virtual histological staining of unlabeled autopsy tissue
University of California, Los Angeles（加州大学洛杉矶分校）；Xi’an Jiaotong University（西安交通大学）；University of California, San Francisco（加州大学旧金山分校）；Hadassah Hebrew University Medical Center（哈达萨希伯来大学医学中心）；University of Southern California（南加州大学）
A robust and scalable framework for hallucination detection in virtual tissue staining and digital pathology
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；Hadassah Hebrew University Medical Center（哈达萨希伯来大学医学中心）；University of Southern California（南加利福尼亚大学）
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute, University of California, Los Angeles（加州大学洛杉矶分校加州纳米系统研究所）；David Geffen School of Medicine, University of California, Los Angeles（加州大学洛杉矶分校大卫·格芬医学院）
Deep learning-based virtual staining was developed to introduce image contrast to label-free tissue sections, digitally matching the histological staining, which is time-consuming, labor-intensive, and destructive to tissue. Standard virtual staining requires high autofocusing precision during the whole slide imaging of label-free tissue, which consumes a significant portion of the total imaging time and can lead to tissue photodamage. Here, we introduce a fast virtual staining framework that can stain defocused autofluorescence images of unlabeled tissue, achieving equivalent performance to v…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）
Using a deep neural network, we demonstrate a digital staining technique, which we term PhaseStain, to transform quantitative phase images (QPI) of labelfree tissue sections into images that are equivalent to brightfield microscopy images of the same samples that are histochemically stained. Through pairs of image data (QPI and the corresponding brightfield images, acquired after staining) we train a generative adversarial network (GAN) and demonstrate the effectiveness of this virtual staining approach using sections of human skin, kidney and liver tissue, matching the brightfield microscopy …
Nanyang Technological University（南洋理工大学）；Institute of Bioengineering and Bioimaging, A*STAR（A*STAR生物工程与生物成像研究所）
Staining is critical to cell imaging and medical diagnosis, which is expensive, time-consuming, labor-intensive, and causes irreversible changes to cell tissues. Recent advances in deep learning enabled digital staining via supervised model training. However, it is difficult to obtain large-scale stained/unstained cell image pairs in practice, which need to be perfectly aligned with the supervision. In this work, we propose a novel unsupervised deep learning framework for the digital staining of cell images using knowledge distillation and generative adversarial networks (GANs). A teacher mode…
Score-based Diffusion Model for Unpaired Virtual Histology Staining
Department of Health Technology and Informatics, Hong Kong Polytechnic University, China（香港理工大学健康科技与信息学系）；Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；School of Science and Engineering, University of Dundee, UK（邓迪大学科学与工程学院）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK（剑桥大学应用数学与理论物理系）
Virtual birefringence imaging and histological staining of amyloid deposits in label-free tissue using autofluorescence microscopy and deep learning
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI), University of California, Los Angeles（加州大学洛杉矶分校加州纳米系统研究所）；Fatih Sultan Mehmet Vakif University（法提赫·苏丹·穆罕默德·瓦克夫大学）；David Geffen School of Medicine at the University of California, Los Angeles（加州大学洛杉矶分校大卫·格芬医学院）；Hadassah Hebrew University Medical Center（哈达萨希伯来大学医学中心）；Keck School of Medicine, University of Southern California（南加州大学凯克医学院）
Histology Virtual Staining with Mask-Guided Adversarial Transfer Learning for Tertiary Lymphoid Structure Detection
Yu-Yue Pathology Research Center（渝越病理研究中心）；Jinfeng Laboratory（金凤实验室）；Institute of Pathology and Southwest Cancer Center, Southwest Hospital, Army Medical University (Third Military Medical University)（陆军军医大学（第三军医大学）西南医院病理科及西南癌症中心）
Pathological Semantics-Preserving Learning for H&E-to-IHC Virtual Staining
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences（中国科学院大学深圳先进技术学院）；The Hong Kong Polytechnic University（香港理工大学）；School of Computer Science, University of Nottingham Ningbo China（宁波诺丁汉大学计算机学院）
Computer-aided diagnosis (CAD) can be used as an important tool to aid and enhance pathologists' diagnostic decision-making. Deep learning techniques, such as convolutional neural networks (CNN) and fully convolutional networks (FCN), have been successfully applied in medical and biological research. Unfortunately, histological image segmentation is often constrained by the availability of labeled training data once labeling histological images for segmentation purposes is a highly-skilled, complex, and time-consuming task. This paper presents the hematoxylin and eosin (H&E) stained oral cavit…
Digital Staining with Knowledge Distillation: A Unified Framework for Unpaired and Paired-But-Misaligned Data
Nanyang Technological University（南洋理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Institute of Materials Research and Engineering, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局材料研究与工程研究所）
Unpaired Multi-Domain Histopathology Virtual Staining using Dual Path Prompted Inversion
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；National innovation center for advanced medical devices（国家高性能医疗器械创新中心）
From Fibers to Cells: Fourier-Based Registration Enables Virtual Cresyl Violet Staining From 3D Polarized Light Imaging
Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich（神经科学与医学研究所（INM-1），茹里奇研究中心）；Helmholtz AI, Research Centre Jülich（海德堡人工智能研究所，茹里奇研究中心）；Cécile & Oskar Vogt Institute of Brain Research, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf（塞西尔与奥斯卡·沃格特脑研究研究所，杜塞尔多夫医学院和大学医院，海因里希·海涅大学）；Department of Physics, University of Wuppertal（物理系，乌尔姆大学）；Institute of Computer Science, Faculty of Mathematics and Natural Sciences, Heinrich Heine University Düsseldorf（计算机科学研究所，数学与自然科学学院，海因里希·海涅大学）
Label- and slide-free tissue histology using 3D epi-mode quantitative phase imaging and virtual H&E staining
University of California, Davis（加利福尼亚大学戴维斯分校）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Emory University School of Medicine（埃默里大学医学院）；UC Davis Health（加州大学戴维斯分校医疗系统）
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（戴维·格芬医学院）
Pathological diagnosis relies on the visual inspection of histologically stained thin tissue specimens, where different types of stains are applied to bring contrast to and highlight various desired histological features. However, the destructive histochemical staining procedures are usually irreversible, making it very difficult to obtain multiple stains on the same tissue section. Here, we demonstrate a virtual stain transfer framework via a cascaded deep neural network (C-DNN) to digitally transform hematoxylin and eosin (H&E) stained tissue images into other types of histological stains. U…
Many medical and biological protocols for analyzing individual biological cells involve morphological evaluation based on cell staining, designed to enhance imaging contrast and enable clinicians and biologists to differentiate between various cell organelles. However, cell staining is not always allowed in certain medical procedures. In other cases, staining may be time consuming or expensive to implement. Here, we present a new deep-learning approach, called HoloStain, which converts images of isolated biological cells acquired without staining by holographic microscopy to their virtually st…
Scalable, Trustworthy Generative Model for Virtual Multi-Staining from H&E Whole Slide Images
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共医疗集团）；Inria（法国国家信息与自动化研究所）；Paris Brain Institute-ICM（巴黎大脑研究所-脑研究所）；Paris Cité University（巴黎城市大学）；Necker-Enfants Malades University Hospitals（内克尔-儿童马尔代夫大学医院）；Centre de recherche sur l’inflammation（炎症研究中心）；Robert Debré University Hospitals（罗伯特德布雷大学医院）
A Value Mapping Virtual Staining Framework for Large-scale Histological Imaging
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；National Engineering Research Center of Visual Technology, Peking University（北京大学视觉技术国家工程研究中心）；School of Computer Science, Peking University（北京大学计算机学院）；Medical School, Nanjing University（南京大学医学院）
We propose a virtual staining methodology based on Generative Adversarial Networks to map histopathology images of breast cancer tissue from H&E stain to PHH3 and vice versa. We use the resulting synthetic images to build Convolutional Neural Networks (CNN) for automatic detection of mitotic figures, a strong prognostic biomarker used in routine breast cancer diagnosis and grading. We propose several scenarios, in which CNN trained with synthetically generated histopathology images perform on par with or even better than the same baseline model trained with real images. We discuss the potentia…
Virtual staining for 3D X-ray histology of bone implants
Institute of Materials Physics（材料物理研究所）；Helmholtz-Zentrum Hereon（海德堡研究中心）；Institute of Metallic Biomaterials（金属生物材料研究所）；Bruker Daltonics SPR（布鲁克尔达特利克斯SPR）；Data-Driven Analysis and Design of Materials（数据驱动的材料分析与设计）；University of Rostock（罗斯托克大学）
Pathology-Guided Virtual Staining Metric for Evaluation and Training
Department of Systems Design Engineering（系统设计工程系）；Department of Applied Mathematics（应用数学系）；Cheriton School of Computer Science（切里顿计算机科学学院）；Department of Biology（生物学系）；School of Pharmacy（药学院）
ContiStain: Cross-Domain Relation-Preserving Distillation for Continual Multi-Domain Virtual IHC Staining
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）；Boston University School of Medicine（波士顿大学医学院）；VA Boston Healthcare（波士顿退伍军人医疗中心）
Histopathological diagnoses of tumors in tissue biopsy after Hematoxylin and Eosin (H&E) staining is the gold standard for oncology care. H&E staining is slow and uses dyes, reagents and precious tissue samples that cannot be reused. Thousands of native nonstained RGB Whole Slide Image (RWSI) patches of prostate core tissue biopsies were registered with their H&E stained versions. Conditional Generative Adversarial Neural Networks (cGANs) that automate conversion of native nonstained RWSI to computational H&E stained images were then trained. High similarities between computational and H&E dye…
Staining normalization in histopathology: Method benchmarking using multicenter dataset
University of Turku, Institute of Biomedicine（图尔库大学生物医学研究所）；Department of Pathology, Hospital Nova of Central Finland（中央芬兰医院病理科部）；University of Eastern Finland, Institute of Biomedicine（东部芬兰大学生物医学研究所）；Tampere University, Faculty of Medicine and Health Technology（塔皮奥拉大学医学与健康技术学院）；InFlames Research Flagship, University of Turku（InFlames研究旗舰，图尔库大学）
The University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；The Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Histopathological analysis is crucial in artery characterization for coronary artery disease (CAD). However, histology requires an invasive and time-consuming process. In this paper, we propose to generate virtual histology staining using Optical Coherence Tomography (OCT) images to enable real-time histological visualization. We develop a deep learning network, namely Coronary-GAN, to transfer coronary OCT images to virtual histology images. With a special consideration on the structural constraints in coronary OCT images, our method achieves better image generation performance than the conve…
Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. To acquire color images of histology slides, it often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learni…
HistDiT: A Structure-Aware Latent Conditional Diffusion Model for High-Fidelity Virtual Staining in Histopathology
Edge Hill University（埃奇希尔大学）；University of Nottingham Malaysia（诺丁汉大学马来西亚分校）；University of Southampton Malaysia（南安普顿大学马来西亚分校）；Cancer Research Malaysia（马来西亚癌症研究中心）
University of Strasbourg（斯特拉斯堡大学）；University of Belgrade（贝尔格莱德大学）；University of Kragujevac（克拉古耶瓦茨大学）；Hannover Medical School（汉诺威医学院）；University Clinic, Freiburg（弗赖堡大学诊所）
The application of supervised deep learning methods in digital pathology is limited due to their sensitivity to domain shift. Digital Pathology is an area prone to high variability due to many sources, including the common practice of evaluating several consecutive tissue sections stained with different staining protocols. Obtaining labels for each stain is very expensive and time consuming as it requires a high level of domain knowledge. In this article, we propose an unsupervised augmentation approach based on adversarial image-to-image translation, which facilitates the training of stain in…
Histopathological visualizations are a pillar of modern medicine and biological research. Surgical oncology relies exclusively on post-operative histology to determine definitive surgical success and guide adjuvant treatments. The current histology workflow is based on bright-field microscopic assessment of histochemical stained tissues and has some major limitations. For example, the preparation of stained specimens for brightfield assessment requires lengthy sample processing, delaying interventions for days or even weeks. Hence, there is a pressing need for improved histopathology methods. …
GANs vs. Diffusion Models for virtual staining with the HER2match dataset
The Netherlands Cancer Institute, Amsterdam, The Netherlands（荷兰癌症研究所）；Faculty of Engineering, University of Porto, Porto, Portugal（葡萄牙波尔图大学工程学院）；IMP Diagnostics, Porto, Portugal（IMP诊断（葡萄牙波尔图））
Digital staining in optical microscopy using deep learning -- a review
Duke University（杜克大学）；Friedrich-Alexander University (FAU)（弗里德里希-亚历山大大学）；University of Connecticut（康涅狄格大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Yonsei University（延世大学）；Institute of Medical Biotechnology（医学生物技术研究所）
VIMs: Virtual Immunohistochemistry Multiplex staining via Text-to-Stain Diffusion Trained on Uniplex Stains
Kahlert School of Computing, University of Utah（犹他大学卡勒特计算学院）；Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；The Catholic University of Korea College of Medicine（韩国天主教大学医学院）；Huntsman Cancer Institute, University of Utah Health（犹他大学健康亨茨曼癌症研究所）；Department of Pathology, University of Utah（犹他大学病理学系）
Visualizing the details of different cellular structures is of great importance to elucidate cellular functions. However, it is challenging to obtain high quality images of different structures directly due to complex cellular environments. Fluorescence staining is a popular technique to label different structures but has several drawbacks. In particular, label staining is time consuming and may affect cell morphology, and simultaneous labels are inherently limited. This raises the need of building computational models to learn relationships between unlabeled microscopy images and labeled fluo…
The analysis of FFPE tissue sections stained with haematoxylin and eosin (H&E) or immunohistochemistry (IHC) is an essential part of the pathologic assessment of surgically resected breast cancer specimens. IHC staining has been broadly adopted into diagnostic guidelines and routine workflows to manually assess status and scoring of several established biomarkers, including ER, PGR, HER2 and KI67. However, this is a task that can also be facilitated by computational pathology image analysis methods. The research in computational pathology has recently made numerous substantial advances, often …
Diffusion-based Virtual Staining from Polarimetric Mueller Matrix Imaging
The Hong Kong University of Science and Technology（香港科技大学）；Beijing Institute of Collaborative Innovation（北京协同创新研究院）；Peking University Health Science Center（北京大学医学部）；University of Oxford（牛津大学）；Peking University（北京大学）
Stain color variation in histological images, caused by a variety of factors, is a challenge not only for the visual diagnosis of pathologists but also for cell segmentation algorithms. To eliminate the color variation, many stain normalization approaches have been proposed. However, most were designed for hematoxylin and eosin staining images and performed poorly on immunohistochemical staining images. Current cell segmentation methods systematically apply stain normalization as a preprocessing step, but the impact brought by color variation has not been quantitatively investigated yet. In th…
Generative Adversarial Networks (GANs) are state of the art for image synthesis. Here, we present dapi2ck, a novel GAN-based approach to synthesize cytokeratin (CK) staining from immunofluorescent (IF) DAPI staining of nuclei in non-small cell lung cancer (NSCLC) images. We use the synthetic CK to segment epithelial regions, which, compared to expert annotations, yield equally good results as segmentation on stained CK. Considering the limited number of markers in a multiplexed IF (mIF) panel, our approach allows to replace CK by another marker addressing the complexity of the tumor micro-envi…
NCIS: Deep Color Gradient Maps Regression and Three-Class Pixel Classification for Enhanced Neuronal Cell Instance Segmentation in Nissl-Stained Histological Images
London South Bank University（伦敦南岸大学）；University of Padova（帕多瓦大学）
The diagnosis of cancer is mainly performed by visual analysis of the pathologists, through examining the morphology of the tissue slices and the spatial arrangement of the cells. If the microscopic image of a specimen is not stained, it will look colorless and textured. Therefore, chemical staining is required to create contrast and help identify specific tissue components. During tissue preparation due to differences in chemicals, scanners, cutting thicknesses, and laboratory protocols, similar tissues are usually varied significantly in appearance. This diversity in staining, in addition to…
Joint analysis of multiple biomarker images and tissue morphology is important for disease diagnosis, treatment planning and drug development. It requires cross-staining comparison among Whole Slide Images (WSIs) of immuno-histochemical and hematoxylin and eosin (H&E) microscopic slides. However, automatic, and fast cross-staining alignment of enormous gigapixel WSIs at single-cell precision is challenging. In addition to morphological deformations introduced during slide preparation, there are large variations in cell appearance and tissue morphology across different staining. In this paper, …
While hematoxylin and eosin (H&E) is a standard staining procedure, immunohistochemistry (IHC) staining further serves as a diagnostic and prognostic method. However, acquiring special staining results requires substantial costs. Hence, we proposed a strategy for ultra-high-resolution unpaired image-to-image translation: Kernelized Instance Normalization (KIN), which preserves local information and successfully achieves seamless stain transformation with constant GPU memory usage. Given a patch, corresponding position, and a kernel, KIN computes local statistics using convolution operation. In…
South China University of Technology（华南理工大学）；Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences（广东省人民医院（广东省医学科学院））；Southern Medical University（南方医科大学）
Color inconsistency is an inevitable challenge in computational pathology, which generally happens because of stain intensity variations or sections scanned by different scanners. It harms the pathological image analysis methods, especially the learning-based models. A series of approaches have been proposed for stain normalization. However, most of them are lack flexibility in practice. In this paper, we formulated stain normalization as a digital re-staining process and proposed a self-supervised learning model, which is called RestainNet. Our network is regarded as a digital restainer which…
