We have developed an automatic method for segmenting fluorescence lifetime (FLT) imaging microscopy (FLIM) images of cells inspired by a multi-resolution community detection (MCD) based network segmentation method. The image processing problem is framed as identifying segments with respective average FLTs against a background in FLIM images. The proposed method segments a FLIM image for a given resolution of the network composed using image pixels as the nodes and similarity between the pixels as the edges. In the resulting segmentation, low network resolution leads to larger segments and high network resolution leads to smaller segments. Further, the mean-square error (MSE) in estimating the FLT segments in a FLIM image using the proposed method was found to be consistently decreasing with increasing resolution of the corresponding network. The proposed MCD method outperformed a popular spectral clustering based method in performing FLIM image segmentation. The spectral segmentation method introduced noisy segments in its output at high resolution. It was unable to offer a consistent decrease in MSE with increasing resolution.
Fluorescence microscopy has enabled a dramatic development in modern biology. Due to its inherently weak signal, fluorescence microscopy is not only much noisier than photography, but also presented with Poisson-Gaussian noise where Poisson noise, or shot noise, is the dominating noise source. To get clean fluorescence microscopy images, it is highly desirable to have effective denoising algorithms and datasets that are specifically designed to denoise fluorescence microscopy images. While such algorithms exist, no such datasets are available. In this paper, we fill this gap by constructing a …
Fluorescence microscopy has become a widely used tool for studying various biological structures of in vivo tissue or cells. However, quantitative analysis of these biological structures remains a challenge due to their complexity which is exacerbated by distortions caused by lens aberrations and light scattering. Moreover, manual quantification of such image volumes is an intractable and error-prone process, making the need for automated image analysis methods crucial. This paper describes a segmentation method for tubular structures in fluorescence microscopy images using convolutional neura…
Compressed sensing fluorescence microscopy (CS-FM) proposes a scheme whereby less measurements are collected during sensing and reconstruction is performed to recover the image. Much work has gone into optimizing the sensing and reconstruction portions separately. We propose a method of jointly optimizing both sensing and reconstruction end-to-end under a total measurement constraint, enabling learning of the optimal sensing scheme concurrently with the parameters of a neural network-based reconstruction network. We train our model on a rich dataset of confocal, two-photon, and wide-field micr…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California Nano Systems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Three-dimensional (3D) fluorescence microscopy in general requires axial scanning to capture images of a sample at different planes. Here we demonstrate that a deep convolutional neural network can be trained to virtually refocus a 2D fluorescence image onto user-defined 3D surfaces within the sample volume. With this data-driven computational microscopy framework, we imaged the neuron activity of a Caenorhabditis elegans worm in 3D using a time-sequence of fluorescence images acquired at a single focal plane, digitally increasing the depth-of-field of the microscope by 20-fold without any axi…
Czech Academy of Sciences（捷克科学院）；Institute of Information Theory and Automation（信息理论与自动化研究所）；Washington University in St. Louis（圣路易斯华盛顿大学）；Washington University School of Medicine（华盛顿大学医学院）
Three-dimensional fluorescence microscopy often suffers from anisotropy, where the resolution along the axial direction is lower than that within the lateral imaging plane. We address this issue by presenting Dual-Cycle, a new framework for joint deconvolution and fusion of dual-view fluorescence images. Inspired by the recent Neuroclear method, Dual-Cycle is designed as a cycle-consistent generative network trained in a self-supervised fashion by combining a dual-view generator and prior-guided degradation model. We validate Dual-Cycle on both synthetic and real data showing its state-of-the-…
University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密西根学院）；Shanghai Jiao Tong University（上海交通大学）
Fluorescence microscopy plays an important role in biomedical research. The depth-variant point spread function (PSF) of a fluorescence microscope produces low-quality images especially in the out-of-focus regions of thick specimens. Traditional deconvolution to restore the out-of-focus images is usually insufficient since a depth-invariant PSF is assumed. This article aims at handling fluorescence microscopy images by learning-based depth-variant PSF and reducing artifacts. We propose adaptive weighting depth-variant deconvolution (AWDVD) with defocus level prediction convolutional neural net…
Robust and accurate nuclei centroid detection is important for the understanding of biological structures in fluorescence microscopy images. Existing automated nuclei localization methods face three main challenges: (1) Most of object detection methods work only on 2D images and are difficult to extend to 3D volumes; (2) Segmentation-based models can be used on 3D volumes but it is computational expensive for large microscopy volumes and they have difficulty distinguishing different instances of objects; (3) Hand annotated ground truth is limited for 3D microscopy volumes. To address these iss…
Multiple stains are usually used to highlight biological substances in biomedical image analysis. To decompose multiple stains for co-localization quantification, blind source separation is usually performed. Prior model-based stain separation methods usually rely on stains' spatial distributions over an image and may fail to solve the co-localization problem. With the advantage of machine learning, deep generative models are used for this purpose. Since prior knowledge of imaging models is ignored in purely data-driven solutions, these methods may be sub-optimal. In this study, a novel learni…
This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. Our results show that by including a model of illumination within the first layers of a deep convolutional neural network, it is possible to learn task-specific LED patterns that substantially improve the ability to infer fluorescence image information from unstained transmission microscopy images. We validated our method on two different experimental setups, with different magnifications and different sample types, to show a consistent improvement in performance as compared to conventional…
Saudi Electronic University（沙特电子大学）；The Commonwealth Scientific and Industrial Research Organisation（澳大利亚联邦科学与工业研究组织）；The Australian National University（澳大利亚国立大学）；University of Technology Sydney（悉尼科技大学）；the University of Western Australia（西澳大利亚大学）
Accurate localization of proteins from fluorescence microscopy images is challenging due to the inter-class similarities and intra-class disparities introducing grave concerns in addressing multi-class classification problems. Conventional machine learning-based image prediction pipelines rely heavily on pre-processing such as normalization and segmentation followed by hand-crafted feature extraction to identify useful, informative, and application-specific features. Here, we demonstrate that deep learning-based pipelines can effectively classify protein images from different datasets. We prop…
Fluorescence microscopy is a quintessential tool for observing cells and understanding the underlying mechanisms of life-sustaining processes of all living organisms. The problem of extracting 3D shape of mitochondria from fluorescence microscopy images remains unsolved due to the complex and varied shapes expressed by mitochondria and the poor resolving capacity of these microscopes. We propose an approach to bridge this gap by learning a shape prior for mitochondria termed as MiShape, by leveraging high-resolution electron microscopy data. MiShape is a generative model learned using implicit…
In recent years, an enormous amount of fluorescence microscopy images were collected in high-throughput lab settings. Analyzing and extracting relevant information from all images in a short time is almost impossible. Detecting tiny individual cell compartments is one of many challenges faced by biologists. This paper aims at solving this problem by building an end-to-end process that employs methods from the deep learning field to automatically segment, detect and classify cell compartments of fluorescence microscopy images of yeast cells. With this intention we used Mask R-CNN to automatical…
The trade-off between throughput and image quality is an inherent challenge in microscopy. To improve throughput, compressive imaging under-samples image signals; the images are then computationally reconstructed by solving a regularized inverse problem. Compared to traditional regularizers, Deep Learning based methods have achieved greater success in compression and image quality. However, the information loss in the acquisition process sets the compression bounds. Further improvement in compression, without compromising the reconstruction quality is thus a challenge. In this work, we propose…
Max Planck Institute of Molecular Cell Biology and Genetics（马克斯·普朗克分子细胞生物学与遗传学研究所）；Center for Systems Biology Dresden（德累斯顿系统生物学中心）；Faculty of Computer Science, Technical University Dresden（德累斯顿工业大学计算机科学系）
Fluorescence microscopy images usually show severe anisotropy in axial versus lateral resolution. This hampers downstream processing, i.e. the automatic extraction of quantitative biological data. While deconvolution methods and other techniques to address this problem exist, they are either time consuming to apply or limited in their ability to remove anisotropy. We propose a method to recover isotropic resolution from readily acquired anisotropic data. We achieve this using a convolutional neural network that is trained end-to-end from the same anisotropic body of data we later apply the net…
ETH Zurich（苏黎世联邦理工学院）；University Hospital（大学医院）；University of Zurich（苏黎世大学）；Uppsala University（乌普萨拉大学）
Fluorescence microscopy images contain several channels, each indicating a marker staining the sample. Since many different marker combinations are utilized in practice, it has been challenging to apply deep learning based segmentation models, which expect a predefined channel combination for all training samples as well as at inference for future application. Recent work circumvents this problem using a modality attention approach to be effective across any possible marker combination. However, for combinations that do not exist in a labeled training dataset, one cannot have any estimation of…
Compressive Macroscopic Fluorescence Lifetime Imaging (MFLI) is a novel technical implementation that enables monitoring multiple molecular interactions in macroscopic scale. Especially, we reported recently on the development of a hyperspectral wide-field time-resolved single-pixel imaging platform that facilitates whole-body in vivo lifetime imaging in less than 14 minutes. However, despite efficient data acquisition, the data processing of a Compressed Sensing (CS) based inversion plus lifetime fitting remain very time consuming. Herein, we propose to investigate the potential of deep learn…
We present a dual color laser scanning endomicroscope capable of fluorescence lifetime endomicroscopy at one frame per second (FPS). The scanning system uses a coherent imaging fiber with 30,000 cores. High-speed lifetime imaging is achieved by distributing the signal over an array of 1024 parallel single-photon avalanche photo--diode detectors (SPADs), minimizing detection dead time maximizing the number of photons detected per excitation pulse without photon pile--up to achieve the high frame rate. Dual color fluorescence imaging is achieved by temporally shifting the dual excitation lasers,…
Fluorescence microscopy has revolutionized biomedical research over the past three decades. Its high molecular specificity and unrivaled single molecule level sensitivity have enabled breakthroughs in a variety of research fields. For in vivo applications, its major limitation is the superficial imaging depth as random scattering in biological tissues causes exponential attenuation of the ballistic component of a light wave. Here we present fluorescence microscopy beyond the ballistic regime by combining single cycle pulsed ultrasound modulation and digital optical phase conjugation. We demons…
The efficient optical tool for elimination of the phase and amplitude distortions produced by imperfectness of the optical elements in microscope lightpath considered. This robust procedure described by simple theoretical model proved to be successful to repair the noisy images with noise to signal ratio close to 100. It is shown that Van Cittert - Zernike theorem provides adequate description of imperfect microscope operating under both coherent and incoherent illumination conditions and having turbid media in object and intermediate planes.
Phantom eye devices are designed to mimic the optical properties of the human eye; however, the retinal structures are typically emulated with synthetic tissues. Thus, it is currently impossible to exploit phantom eye devices for fluorescence microscopy experiments on cultures, or ex-vivo tissues, although these ophthalmic measurements would be of importance for many applications such as the testing of retinal biomarkers. Here we report and describe the development of a phantom eye designed to host biological samples, such as retinal cultures differentiated from human induced pluripotent stem …
The aberrations induced by the sample and/or by the sample holder limit the resolution of optical microscopes. Wavefront correction can be achieved using a deformable mirror with wavefront sensorless optimization algorithms but, despite the complexity of these systems, the level of correction is often limited to a small area in the field of view of the microscope. In this work, we present a plug and play module for aberration measurement and correction. The wavefront correction is performed through direct wavefront reconstruction using the spinning-pupil aberration measurement and controlling …
Imaging of rapidly occurring events in biology requires high temporal resolution. In this review article, I have tried to investigate an approximate estimate to quantify temporal resolution limit of a fluorescence imaging system. It was realised that, the temporal resolution is essentially determined by the fact that, once excited, almost all ($99.9\%$) excited molecules relax to ground state. We determined the time required for $99.9\%$ of molecules to relax is about $3τ_p = 3/{(k_f +k_{nr})\log_{10}e}$, where $k_f +k_{nr}$ is the total emission rate. To arrive at this expression we assumed t…
We report on the recently emerging (Laser) Light Sheet based Fluorescence Microscopy field (LSFM). The techniques used in this field allow to study and visualize biomedical objects non-destructively in high-resolution through virtual optical sectioning with sheets of laser light. Fluorescence originating in the cross section of the sheet and sample is recorded orthogonally with a camera. In this paper, the first implementation of LSFM to image biomedical tissue in three dimensions - Orthogonal-Plane Fluorescence Optical Sectioning microscopy (OPFOS) - is discussed. Since then many similar and …
PyFLI: A Python Library for Simulation, Parameter Estimation, and Benchmarking in Fluorescence Lifetime Imaging
Center for Modeling, Simulation and Imaging in Medicine, Rensselaer Polytechnic Institute（医学建模、模拟与成像中心，伦斯勒理工学院）；Department of Molecular and Cellular Physiology, Albany Medical College（分子与细胞生理学系，奥尔巴尼医学院）
