We demonstrate a deep learning-based offline autofocusing method, termed Deep-R, that is trained to rapidly and blindly autofocus a single-shot microscopy image of a specimen that is acquired at an arbitrary out-of-focus plane. We illustrate the efficacy of Deep-R using various tissue sections that were imaged using fluorescence and brightfield microscopy modalities and demonstrate snapshot autofocusing under different scenarios, such as a uniform axial defocus as well as a sample tilt within the field-of-view. Our results reveal that Deep-R is significantly faster when compared with standard online algorithmic autofocusing methods. This deep learning-based blind autofocusing framework opens up new opportunities for rapid microscopic imaging of large sample areas, also reducing the photon dose on the sample.
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；McMaster University（麦克马斯特大学）；Shanghai Jiao Tong University（上海交通大学）
Whole slide imaging (WSI) is an emerging technology for digital pathology. The process of autofocusing is the main influence of the performance of WSI. Traditional autofocusing methods either are time-consuming due to repetitive mechanical motions, or require additional hardware and thus are not compatible to current WSI systems. In this paper, we propose the concept of \textit{virtual autofocusing}, which does not rely on mechanical adjustment to conduct refocusing but instead recovers in-focus images in an offline learning-based manner. With the initial focal position, we only perform two-sh…
Whole slide imaging (WSI) has moved digital pathology closer to diagnostic practice in recent years. Due to the inherent tissue topography variability, accurate autofocusing remains a critical challenge for WSI and automated microscopy systems. The traditional focus map surveying method is limited in its ability to acquire a high degree of focus points while still maintaining high throughput. Real-time approaches decouple image acquisition from focusing, thus allowing for rapid scanning while maintaining continuous accurate focus. This work reviews the traditional focus map approach and discus…
The numerical wavefront backpropagation principle of digital holography confers unique extended focus capabilities, without mechanical displacements along z-axis. However, the determination of the correct focusing distance is a non-trivial and time consuming issue. A deep learning (DL) solution is proposed to cast the autofocusing as a regression problem and tested over both experimental and simulated holograms. Single wavelength digital holograms were recorded by a Digital Holographic Microscope (DHM) with a 10$\mathrm{x}$ microscope objective from a patterned target moving in 3D over an axia…
A Zero-Shot Deep Image Prior Framework for Denoising and Deconvolution in Fluorescence Microscopy
Wenzhou Key Laboratory of Biomedical Imaging（温州生物医学成像重点实验室）；Zhejiang Key Laboratory of Soft Matter Biomedical Materials（浙江软物质生物医学材料重点实验室）；Wenzhou Institute, University of Chinese Academy of Sciences（中国科学院大学温州研究院）；Postgraduate Training Base Alliance of Wenzhou Medical University（温州医学院研究生联合培养基地联盟）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California Nano Systems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine, University of California, Los Angeles（加利福尼亚大学洛杉矶分校大卫·格芬医学院）
Digital holography is one of the most widely used label-free microscopy techniques in biomedical imaging. Recovery of the missing phase information of a hologram is an important step in holographic image reconstruction. Here we demonstrate a convolutional recurrent neural network (RNN) based phase recovery approach that uses multiple holograms, captured at different sample-to-sensor distances to rapidly reconstruct the phase and amplitude information of a sample, while also performing autofocusing through the same network. We demonstrated the success of this deep learning-enabled holography me…
Universität Ulm（乌尔姆大学）；Ulm University Medical Center（乌尔姆大学医学中心）；The University of Adelaide（阿德莱德大学）；Otto von Guericke University（奥托·冯·格里克大学）
Deep neural networks currently deliver promising results for microscopy image cell segmentation, but they require large-scale labelled databases, which is a costly and time-consuming process. In this work, we relax the labelling requirement by combining self-supervised with semi-supervised learning. We propose the prediction of edge-based maps for self-supervising the training of the unlabelled images, which is combined with the supervised training of a small number of labelled images for learning the segmentation task. In our experiments, we evaluate on a few-shot microscopy image cell segmen…
Universität Ulm（乌尔姆大学）；The University of Adelaide（阿德莱德大学）
Automatic cell segmentation in microscopy images works well with the support of deep neural networks trained with full supervision. Collecting and annotating images, though, is not a sustainable solution for every new microscopy database and cell type. Instead, we assume that we can access a plethora of annotated image data sets from different domains (sources) and a limited number of annotated image data sets from the domain of interest (target), where each domain denotes not only different image appearance but also a different type of cell segmentation problem. We pose this problem as meta-l…
Background: The use of whole slide imaging (WSI) for digital pathology has recently been cleared for primary diagnosis in the US. A conventional WSI system scans the tissue slide to different positions and acquires the digital images. In a typical implementation, a focus map is created prior to the scanning process, leading to significant overhead time and a necessity for high positional accuracy of the mechanical system. The resulting cost of WSI system is often prohibitive for frozen section procedure during surgery. Methods: We report a novel WSI scheme based on a programmable LED array for…
Universität Ulm（乌尔姆大学）；Mercedes-Benz AG（梅赛德斯-奔驰股份公司）；Ulm University Medical Center（乌尔姆大学医学中心）；Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学视觉、语音与信号处理中心）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
In microscopy image cell segmentation, it is common to train a deep neural network on source data, containing different types of microscopy images, and then fine-tune it using a support set comprising a few randomly selected and annotated training target images. In this paper, we argue that the random selection of unlabelled training target images to be annotated and included in the support set may not enable an effective fine-tuning process, so we propose a new approach to optimise this image selection process. Our approach involves a new scoring function to find informative unlabelled target…
Fluorescence microscopy has enabled a dramatic development in modern biology. Due to its inherently weak signal, fluorescence microscopy is not only much noisier than photography, but also presented with Poisson-Gaussian noise where Poisson noise, or shot noise, is the dominating noise source. To get clean fluorescence microscopy images, it is highly desirable to have effective denoising algorithms and datasets that are specifically designed to denoise fluorescence microscopy images. While such algorithms exist, no such datasets are available. In this paper, we fill this gap by constructing a …
Light-field microscopes are able to capture spatial and angular information of incident light rays. This allows reconstructing 3D locations of neurons from a single snap-shot.In this work, we propose a model-inspired deep learning approach to perform fast and robust 3D localization of sources using light-field microscopy images. This is achieved by developing a deep network that efficiently solves a convolutional sparse coding (CSC) problem to map Epipolar Plane Images (EPI) to corresponding sparse codes. The network architecture is designed systematically by unrolling the convolutional Iterat…
University of Copenhagen（哥本哈根大学）；Rigshospitalet（哥本哈根大学医院（Rigshospitalet））
Tracking single molecules is instrumental for quantifying the transport of molecules and nanoparticles in biological samples, e.g., in brain drug delivery studies. Existing intensity-based localisation methods are not developed for imaging with a scanning microscope, typically used for in vivo imaging. Low signal-to-noise ratios, movement of molecules out-of-focus, and high motion blur on images recorded with scanning two-photon microscopy (2PM) in vivo pose a challenge to the accurate localisation of molecules. Using data-driven models is challenging due to low data volumes, typical for in vi…
Zero-shot Shape Classification of Nanoparticles in SEM Images using Vision Foundation Models
Department of Bioinformatics, Jerusalem College of Technology（生物信息学系，耶路撒冷技术学院）；Bar-Ilan University（巴伊兰大学）；Jerusalem College of Technology（耶路撒冷技术学院）；Bar-Ilan Institute of Nanotechnology and Advanced Material（巴伊兰纳米技术与先进材料研究所）；Department of Electrical Engineering / Department of Electrooptical Engineering and Applied Physics（电气工程系 / 电光学工程与应用物理学系）
DM-QPMNET: Dual-modality fusion network for cell segmentation in quantitative phase microscopy
College of Computing and Informatics, University of North Carolina at Charlotte, NC, USA（计算与信息学院，北卡罗来纳大学夏洛特分校）；Department of Physics and Optical Science, University of North Carolina at Charlotte, NC, USA（物理与光学科学系，北卡罗来纳大学夏洛特分校）；Center for TAIMing AI, University of North Carolina at Charlotte, NC, USA（TAIMing AI中心，北卡罗来纳大学夏洛特分校）
Structured illumination microscopy (SIM) can double the resolution beyond the light diffraction limit, but it comes at the cost of multiple camera exposures and the heavy computation burden of multiple Fourier transforms. In this paper, we report a novel technique termed single-shot SIM, to overcome these limitations. A multi-task joint deep-learning strategy is proposed. Generative adversative networks (GAN) are employed to generate five structured illumination images based on the single-shot structured illumination image. U-Net is employed to reconstruct the super-resolution image from these…
White light phase-shifting interference microscopy (WL-PSIM) is a prominent technique for high-resolution quantitative phase imaging (QPI) of industrial and biological specimens. However, multiple interferograms with accurate phase-shifts are essentially required in WL-PSIM for measuring the accurate phase of the object. Here, we present single-shot phase-shifting interferometric techniques for accurate phase measurement using filtered white light phase-shifting interference microscopy (F-WL-PSIM) and deep neural network (DNN). The methods are incorporated by training the DNN to generate 1) fo…
Fourier ptychographic microscopy allows for the collection of images with a high space-bandwidth product at the cost of temporal resolution. In Fourier ptychographic microscopy, the light source of a conventional widefield microscope is replaced with a light-emitting diode (LED) matrix, and multiple images are collected with different LED illumination patterns. From these images, a higher-resolution image can be computationally reconstructed without sacrificing field-of-view. We use deep learning to achieve single-shot imaging without sacrificing the space-bandwidth product, reducing the acqui…
Universidad Politécnica de Madrid（马德里理工大学）；Centre national de la recherche scientifique (CNRS)（法国国家科学研究中心）；Interdisciplinary Institute for Neuroscience (IINS)（跨学科神经科学研究所）；University of Bordeaux（波尔多大学）；Mechanobiology Institute（机械生物学研究所）；National University of Singapore（新加坡国立大学）
Microscopy images often suffer from high levels of noise, which can hinder further analysis and interpretation. Content-aware image restoration (CARE) methods have been proposed to address this issue, but they often require large amounts of training data and suffer from over-fitting. To overcome these challenges, we propose a novel framework for few-shot microscopy image denoising. Our approach combines a generative adversarial network (GAN) trained via contrastive learning (CL) with two structure preserving loss terms (Structural Similarity Index and Total Variation loss) to further improve t…
Miniature fluorescence microscopes are a standard tool in systems biology. However, widefield miniature microscopes capture only 2D information, and modifications that enable 3D capabilities increase the size and weight and have poor resolution outside a narrow depth range. Here, we achieve the 3D capability by replacing the tube lens of a conventional 2D Miniscope with an optimized multifocal phase mask at the objective's aperture stop. Placing the phase mask at the aperture stop significantly reduces the size of the device, and varying the focal lengths enables a uniform resolution across a …
Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane. Placing the MLA near the pupil plane of the microscope, instead of the image plane, can mitigate the artifacts and provide an efficient forward model, at the expense of field-of-view (FOV). Here, we demonstrate improved resolution across a large volume with Fourier DiffuserScope, which uses a diffuser in the pupil plan…
Zero-Shot Image Denoising for High-Resolution Electron Microscopy
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Lingang Laboratory（临港实验室）；School of Physical Science and Technology, ShanghaiTech University（上海科技大学物理科学与技术学院）；School of Software, Tsinghua University（清华大学软件学院）；School of Biomedical Engineering and Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学生物医学工程学院与医学机器人研究所）
Nanjing University of Science and Technology（南京理工大学）；Soochow University（苏州大学）
Structured illumination microscopy (SIM) is an important super-resolution based microscopy technique that breaks the diffraction limit and enhances optical microscopy systems. With the development of biology and medical engineering, there is a high demand for real-time and robust SIM imaging under extreme low light and short exposure environments. Existing SIM techniques typically require multiple structured illumination frames to produce a high-resolution image. In this paper, we propose a single-frame structured illumination microscopy (SF-SIM) based on deep learning. Our SF-SIM only needs o…
We propose to use high numerical aperture single mode optical fibers like photonic crystal fiber for lensless in-line holographic microscopy. Highly divergent beam helps to overcome the spatial sampling limitation of the image sensor. In this paper, a submicron lateral resolution has been demonstrated, with an imaging sensor of pixel pitch 1.12 micrometer and a photonic crystal fiber of mode field diameter 1.8 micrometer. In earlier methods of single-shot lensless imaging, submicron resolution has been obtained at very small working distance and field of view. The proposed method improves the …
Fluorescence microscopy plays a vital role in understanding the subcellular structures of living cells. However, it requires considerable effort in sample preparation related to chemical fixation, staining, cost, and time. To reduce those factors, we present a virtual fluorescence staining method based on deep neural networks (VirFluoNet) to transform fluorescence images of molecular labels into other molecular fluorescence labels in the same field-of-view. To achieve this goal, we develop and train a conditional generative adversarial network (cGAN) to perform digital fluorescence imaging dem…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California Nano Systems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）
Deep learning-based methods in computational microscopy have been shown to be powerful but in general face some challenges due to limited generalization to new types of samples and requirements for large and diverse training data. Here, we demonstrate a few-shot transfer learning method that helps a holographic image reconstruction deep neural network rapidly generalize to new types of samples using small datasets. We pre-trained a convolutional recurrent neural network on a large dataset with diverse types of samples, which serves as the backbone model. By fixing the recurrent blocks and tran…
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）
Polarized light microscopy provides high contrast to birefringent specimen and is widely used as a diagnostic tool in pathology. However, polarization microscopy systems typically operate by analyzing images collected from two or more light paths in different states of polarization, which lead to relatively complex optical designs, high system costs or experienced technicians being required. Here, we present a deep learning-based holographic polarization microscope that is capable of obtaining quantitative birefringence retardance and orientation information of specimen from a phase recovered …
University of California, Berkeley（加州大学伯克利分校）；Orange Coast College（奥兰治海岸学院）
Compressive lensless imagers enable novel applications in an extremely compact device, requiring only a phase or amplitude mask placed close to the sensor. They have been demonstrated for 2D and 3D microscopy, single-shot video, and single-shot hyperspectral imaging; in each of these cases, a compressive-sensing-based inverse problem is solved in order to recover a 3D data-cube from a 2D measurement. Typically, this is accomplished using convex optimization and hand-picked priors. Alternatively, deep learning-based reconstruction methods offer the promise of better priors, but require many tho…
In fluorescence microscopy live-cell imaging, there is a critical trade-off between the signal-to-noise ratio and spatial resolution on one side, and the integrity of the biological sample on the other side. To obtain clean high-resolution (HR) images, one can either use microscopy techniques, such as structured-illumination microscopy (SIM), or apply denoising and super-resolution (SR) algorithms. However, the former option requires multiple shots that can damage the samples, and although efficient deep learning based algorithms exist for the latter option, no benchmark exists to evaluate the…
Precisely measuring the three-dimensional position and orientation of individual fluorophores is challenging due to the substantial photon shot noise in single-molecule experiments. Facing this limited photon budget, numerous techniques have been developed to encode 2D and 3D position and 2D and 3D orientation information into fluorescence images. In this work, we adapt classical and quantum estimation theory and propose a mathematical framework to derive the best possible precision for measuring the position and orientation of dipole-like emitters for any fixed imaging system. We find that it…
Idiap Research Institute（IDIAP研究所）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Optical microscopy is an essential tool in biology and medicine. Imaging thin, yet non-flat objects in a single shot (without relying on more sophisticated sectioning setups) remains challenging as the shallow depth of field that comes with high-resolution microscopes leads to unsharp image regions and makes depth localization and quantitative image interpretation difficult. Here, we present a method that improves the resolution of light microscopy images of such objects by locally estimating image distortion while jointly estimating object distance to the focal plane. Specifically, we estimat…
Lensless in-line holography is a simple, portable, and cost-effective method of imaging especially for the biomedical microscopy applications. We propose a multiplicative gradient descent optimization based method to obtain multi-depth imaging from a single hologram acquired in this imaging system. We further extend the method to achieve phase imaging from a single hologram. Negative-log-likelihood functional with the assumption of poisson noise has been used as the cost function to be minimized. The ill-posed nature of the problem is handled by the sparse regularization and the upper-bound co…
Quantitative phase microscopy (QPM) is a label-free technique that enables to monitor morphological changes at subcellular level. The performance of the QPM system in terms of spatial sensitivity and resolution depends on the coherence properties of the light source and the numerical aperture (NA) of objective lenses. Here, we propose high space-bandwidth QPM using partially spatially coherent optical coherence microscopy (PSC-OCM) assisted with deep neural network. The PSC source synthesized to improve the spatial sensitivity of the reconstructed phase map from the interferometric images. Fur…
Enhanced Confocal Laser Scanning Microscopy with Adaptive Physics Informed Deep Autoencoders
Georgia State University（佐治亚州立大学）；Riphah International University（里法国际大学）；Roots IVY（鲁茨艾维）；Arid Agriculture University（干旱农业大学）；University of Central Florida（中佛罗里达大学）
In example-based super-resolution, the function relating low-resolution images to their high-resolution counterparts is learned from a given dataset. This data-driven approach to solving the inverse problem of increasing image resolution has been implemented with deep learning algorithms. In this work, we explore modifying the imaging hardware in order to collect more informative low-resolution images for better ultimate high-resolution image reconstruction. We show that this "physical preprocessing" allows for improved image reconstruction with deep learning in Fourier ptychographic microscop…
University of Gothenburg（哥德堡大学）；Chalmers University of Technology（查尔姆斯理工大学）
Particle tracking is a fundamental task in digital microscopy. Recently, machine-learning approaches have made great strides in overcoming the limitations of more classical approaches. The training of state-of-the-art machine-learning methods almost universally relies on either vast amounts of labeled experimental data or the ability to numerically simulate realistic datasets. However, the data produced by experiments are often challenging to label and cannot be easily reproduced numerically. Here, we propose a novel deep-learning method, named LodeSTAR (Low-shot deep Symmetric Tracking And Re…
