Visualizing the details of different cellular structures is of great importance to elucidate cellular functions. However, it is challenging to obtain high quality images of different structures directly due to complex cellular environments. Fluorescence staining is a popular technique to label different structures but has several drawbacks. In particular, label staining is time consuming and may affect cell morphology, and simultaneous labels are inherently limited. This raises the need of building computational models to learn relationships between unlabeled microscopy images and labeled fluorescence images, and to infer fluorescence labels of other microscopy images excluding the physical staining process. We propose to develop a novel deep model for virtual staining of unlabeled microscopy images. We first propose a novel network layer, known as the global pixel transformer layer, that fuses global information from inputs effectively. The proposed global pixel transformer layer can generate outputs with arbitrary dimensions, and can be employed for all the regular, down-sampling, and up-sampling operators. We then incorporate our proposed global pixel transformer layers and dense blocks to build an U-Net like network. We believe such a design can promote feature reusing between layers. In addition, we propose a multi-scale input strategy to encourage networks to capture features at different scales. We conduct evaluations across various fluorescence image prediction tasks to demonstrate the effectiveness of our approach. Both quantitative and qualitative results show that our method outperforms the state-of-the-art approach significantly. It is also shown that our proposed global pixel transformer layer is useful to improve the fluorescence image prediction results.
Virtual Gram staining of label-free bacteria using darkfield microscopy and deep learning
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；Texas A&M University（德克萨斯农工大学）；Texas A&M Engineering Experiment Station（德克萨斯农工大学工程实验站）；David Geffen School of Medicine（大卫·格芬医学院）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）
Using a deep neural network, we demonstrate a digital staining technique, which we term PhaseStain, to transform quantitative phase images (QPI) of labelfree tissue sections into images that are equivalent to brightfield microscopy images of the same samples that are histochemically stained. Through pairs of image data (QPI and the corresponding brightfield images, acquired after staining) we train a generative adversarial network (GAN) and demonstrate the effectiveness of this virtual staining approach using sections of human skin, kidney and liver tissue, matching the brightfield microscopy …
Digital staining in optical microscopy using deep learning -- a review
Duke University（杜克大学）；Friedrich-Alexander University (FAU)（弗里德里希-亚历山大大学）；University of Connecticut（康涅狄格大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Yonsei University（延世大学）；Institute of Medical Biotechnology（医学生物技术研究所）
Virtual birefringence imaging and histological staining of amyloid deposits in label-free tissue using autofluorescence microscopy and deep learning
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI), University of California, Los Angeles（加州大学洛杉矶分校加州纳米系统研究所）；Fatih Sultan Mehmet Vakif University（法提赫·苏丹·穆罕默德·瓦克夫大学）；David Geffen School of Medicine at the University of California, Los Angeles（加州大学洛杉矶分校大卫·格芬医学院）；Hadassah Hebrew University Medical Center（哈达萨希伯来大学医学中心）；Keck School of Medicine, University of Southern California（南加州大学凯克医学院）
Significance: Histopathological analysis of tissues is an essential tool for grading, staging, diagnosing and resecting cancers and other malignancies. Current histopathological techniques require substantial sample processing prior to staining with hematoxylin and eosin (H&E) dyes, to highlight nuclear and cellular morphology. Sample preparation and staining is resource-intensive and introduces potential for variability during sample preparation. Aim: We present a novel method for direct label-free histopathological assessment of formalin fixed paraffin embedded tissue blocks and thin tissue …
Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. To acquire color images of histology slides, it often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learni…
In the past decades, absorption modalities have emerged as powerful tools for label-free functional and structural imaging of cells and tissues. Many biomolecules present unique absorption spectra providing chromophore-specific information on properties such as chemical bonding, and sample composition. As chromophores absorb photons the absorbed energy is emitted as photons (radiative relaxation) or converted to heat and under specific conditions pressure (non-radiative relaxation). Modalities like fluorescence microscopy may capture radiative relaxation to provide contrast, while modalities l…
Score-based Diffusion Model for Unpaired Virtual Histology Staining
Department of Health Technology and Informatics, Hong Kong Polytechnic University, China（香港理工大学健康科技与信息学系）；Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；School of Science and Engineering, University of Dundee, UK（邓迪大学科学与工程学院）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK（剑桥大学应用数学与理论物理系）
Many medical and biological protocols for analyzing individual biological cells involve morphological evaluation based on cell staining, designed to enhance imaging contrast and enable clinicians and biologists to differentiate between various cell organelles. However, cell staining is not always allowed in certain medical procedures. In other cases, staining may be time consuming or expensive to implement. Here, we present a new deep-learning approach, called HoloStain, which converts images of isolated biological cells acquired without staining by holographic microscopy to their virtually st…
From Pixels to Pathology: Restoration Diffusion for Diagnostic-Consistent Virtual IHC
organization= Institute of Pathology, Technical University of Munich, TUM School of Medicine；organization= Computer Aided Medical Procedures (CAMP), TU Munich , city= Munich , country= Germany；organization= Munich Center for Machine Learning (MCML) , city= Munich , country= Germany；organization= TUM School of Computation, Information；organization= Department of Medical Informatics, School of Medicine, Mashhad University of Medical Sciences , city= Mashhad , country= Iran；organization= Munich Data Science Institute (MDSI) , city= Munich , country= Germany
Current methods for diagnosing the progression of multiple types of cancer within patients rely on interpreting stained needle biopsies. This process is time-consuming and susceptible to error throughout the paraffinization, Hematoxylin and Eosin (H&E) staining, deparaffinization, and annotation stages. Fourier Transform Infrared (FTIR) imaging has been shown to be a promising alternative to staining for appropriately annotating biopsy cores without the need for deparaffinization or H&E staining with the use of Fourier Transform Infrared (FTIR) images when combined with machine learning to int…
Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）；Boston University School of Medicine（波士顿大学医学院）；VA Boston Healthcare（波士顿退伍军人医疗中心）
Histopathological diagnoses of tumors in tissue biopsy after Hematoxylin and Eosin (H&E) staining is the gold standard for oncology care. H&E staining is slow and uses dyes, reagents and precious tissue samples that cannot be reused. Thousands of native nonstained RGB Whole Slide Image (RWSI) patches of prostate core tissue biopsies were registered with their H&E stained versions. Conditional Generative Adversarial Neural Networks (cGANs) that automate conversion of native nonstained RWSI to computational H&E stained images were then trained. High similarities between computational and H&E dye…
Label- and slide-free tissue histology using 3D epi-mode quantitative phase imaging and virtual H&E staining
University of California, Davis（加利福尼亚大学戴维斯分校）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Emory University School of Medicine（埃默里大学医学院）；UC Davis Health（加州大学戴维斯分校医疗系统）
Virtual staining for 3D X-ray histology of bone implants
Institute of Materials Physics（材料物理研究所）；Helmholtz-Zentrum Hereon（海德堡研究中心）；Institute of Metallic Biomaterials（金属生物材料研究所）；Bruker Daltonics SPR（布鲁克尔达特利克斯SPR）；Data-Driven Analysis and Design of Materials（数据驱动的材料分析与设计）；University of Rostock（罗斯托克大学）
Fluorescence microscopy plays a vital role in understanding the subcellular structures of living cells. However, it requires considerable effort in sample preparation related to chemical fixation, staining, cost, and time. To reduce those factors, we present a virtual fluorescence staining method based on deep neural networks (VirFluoNet) to transform fluorescence images of molecular labels into other molecular fluorescence labels in the same field-of-view. To achieve this goal, we develop and train a conditional generative adversarial network (cGAN) to perform digital fluorescence imaging dem…
LGD-Net: Latent-Guided Dual-Stream Network for HER2 Scoring with Task-Specific Domain Knowledge
School of Mathematics and Statistics, Fujian Normal University（福建师范大学数学与统计学学院）；Department of Oncology, Mengchao Hepatobiliary Hospital of Fujian Medical University（福建医科大学 Mengchao 肝胆医院肿瘤科）
Sabanci University（萨班哲大学）；ARM Ltd.（ARM有限公司）；University of Zurich（苏黎世大学）；University of Utah（犹他大学）；Izmir University of Economics（伊兹密尔经济大学）；University of Rochester（罗切斯特大学）
Data driven segmentation is an important initial step of shape prior-based segmentation methods since it is assumed that the data term brings a curve to a plausible level so that shape and data terms can then work together to produce better segmentations. When purely data driven segmentation produces poor results, the final segmentation is generally affected adversely. One challenge faced by many existing data terms is due to the fact that they consider only pixel intensities to decide whether to assign a pixel to the foreground or to the background region. When the distributions of the foregr…
ETH Zurich（苏黎世联邦理工学院）；University Hospital（大学医院）；University of Zurich（苏黎世大学）；Uppsala University（乌普萨拉大学）
Fluorescence microscopy images contain several channels, each indicating a marker staining the sample. Since many different marker combinations are utilized in practice, it has been challenging to apply deep learning based segmentation models, which expect a predefined channel combination for all training samples as well as at inference for future application. Recent work circumvents this problem using a modality attention approach to be effective across any possible marker combination. However, for combinations that do not exist in a labeled training dataset, one cannot have any estimation of…
Current pathology workflow involves staining of thin tissue slices, which otherwise would be transparent, followed by manual investigation under the microscope by a trained pathologist. While the hematoxylin and eosin (H&E) stain is well-established and a cost-effective method for visualizing histology slides, its color variability across preparations and subjectivity across clinicians remain unaddressed challenges. To mitigate these challenges, recently we have demonstrated that spatial light interference microscopy (SLIM) can provide a path to intrinsic, objective markers, that are independe…
ETH Zurich（苏黎世联邦理工学院）；University Hospital and University of Zurich（苏黎世大学医院和苏黎世大学）；Uppsala University（乌普萨拉大学）
Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers…
Compressive sensing can overcome the Nyquist criterion and record images with a fraction of the usual number of measurements required. However, conventional measurement bases are susceptible to diffraction and scattering, prevalent in high-resolution microscopy. Here, we explore the random Morlet basis as an optimal set for compressive multiphoton imaging, based on its ability to minimise the space-frequency uncertainty. We implement this approach for the newly developed method of wide-field multiphoton microscopy with single-pixel detection (TRAFIX), which allows imaging through turbid media …
Massachusetts Institute of Technology（麻省理工学院）；Harvard University（哈佛大学）；HHMI Janelia Research Campus（霍华德·休斯医学研究所珍妮亚研究园区）
Single-beam scanning electron microscopes (SEM) are widely used to acquire massive data sets for biomedical study, material analysis, and fabrication inspection. Datasets are typically acquired with uniform acquisition: applying the electron beam with the same power and duration to all image pixels, even if there is great variety in the pixels' importance for eventual use. Many SEMs are now able to move the beam to any pixel in the field of view without delay, enabling them, in principle, to invest their time budget more effectively with non-uniform imaging. In this paper, we show how to use d…
Two-Photon Microscopy (TPM) can provide three-dimensional morphological and functional contrast in vivo. Through proper staining, TPM can be utilized to create virtual, H&E equivalent images and thus can improve throughput in histology-based applications. We previously reported on a new light source for TPM that employs a compact and robust fiber-amplified, directly modulated laser. This laser is pulse-to-pulse wavelength switchable between 1064 nm, 1122 nm, and 1186 nm with an adjustable pulse duration from 50ps to 5ns and arbitrary repetition rates up to 1MHz at kW-peak powers. Despite the l…
Colocalization analysis aims to study complex spatial associations between bio-molecules via optical imaging techniques. However, existing colocalization analysis workflows only assess an average degree of colocalization within a certain region of interest and ignore the unique and valuable spatial information offered by microscopy. In the current work, we introduce a new framework for colocalization analysis that allows us to quantify colocalization levels at each individual location and automatically identify pixels or regions where colocalization occurs. The framework, referred to as spatia…
We report an angle-tilted, wavelength-multiplexed ptychographic modulation approach for multispectral lensless on-chip microscopy. In this approach, we illuminate the specimen with lights at 5 wavelengths simultaneously. A prism is added at the illumination path for spectral dispersion. Lightwaves at different wavelengths, thus, hit the specimen at slightly different incident angles, breaking the ambiguities in mixed state ptychographic reconstruction. At the detection path, we place a thin diffuser in-between the specimen and the monochromatic image sensor for encoding the spectral informatio…
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（戴维·格芬医学院）
Pathological diagnosis relies on the visual inspection of histologically stained thin tissue specimens, where different types of stains are applied to bring contrast to and highlight various desired histological features. However, the destructive histochemical staining procedures are usually irreversible, making it very difficult to obtain multiple stains on the same tissue section. Here, we demonstrate a virtual stain transfer framework via a cascaded deep neural network (C-DNN) to digitally transform hematoxylin and eosin (H&E) stained tissue images into other types of histological stains. U…
Accurate segmentation of 3-D cell nuclei in microscopy images is essential for the study of nuclear organization, gene expression, and cell morphodynamics. Current image segmentation methods are challenged by the complexity and variability of microscopy images and often over-segment or under-segment the cell nuclei. Thus, there is a need to improve segmentation accuracy and reliability, as well as the level of automation. In this paper, we propose a new automated algorithm for robust segmentation of 3-D cell nuclei using the concepts of random walk, graph theory, and mathematical morphology as…
Arizona State University（亚利桑那州立大学）；Barrow Neurological Institute（巴罗神经学研究所）；St. Joseph’s Hospital and Medical Center（圣约瑟夫医院和医疗中心）
Confocal laser endomicroscopy (CLE) allow on-the-fly in vivo intraoperative imaging in a discreet field of view, especially for brain tumors, rather than extracting tissue for examination ex vivo with conventional light microscopy. Fluorescein sodium-driven CLE imaging is more interactive, rapid, and portable than conventional hematoxylin and eosin (H&E)-staining. However, it has several limitations: CLE images may be contaminated with artifacts (motion, red blood cells, noise), and neuropathologists are mainly trained on colorful stained histology slides like H&E while the CLE images are gray…
Investigation of cell structure is hardly imaginable without bright-field microscopy. Numerous modifications such as depth-wise scanning or videoenhancement make this method being state-of-the-art. This raises a question what maximal information can be extracted from ordinary (but well acquired) bright-field images in a model-free way. Here we introduce a method of a physically correct extraction of features for each pixel when these features resemble a transparency spectrum. The method is compatible with existent ordinary bright-field microscopes and requires mathematically sophisticated data…
