We present a novel deep learning approach to reconstruct confocal microscopy stacks from single light field images. To perform the reconstruction, we introduce the LFMNet, a novel neural network architecture inspired by the U-Net design. It is able to reconstruct with high-accuracy a 112x112x57.6$μm^3$ volume (1287x1287x64 voxels) in 50ms given a single light field image of 1287x1287 pixels, thus dramatically reducing 720-fold the time for confocal scanning of assays at the same volumetric resolution and 64-fold the required storage. To prove the applicability in life sciences, our approach is evaluated both quantitatively and qualitatively on mouse brain slices with fluorescently labelled blood vessels. Because of the drastic reduction in scan time and storage space, our setup and method are directly applicable to real-time in vivo 3D microscopy. We provide analysis of the optical design, of the network architecture and of our training procedure to optimally reconstruct volumes for a given target depth range. To train our network, we built a data set of 362 light field images of mouse brain blood vessels and the corresponding aligned set of 3D confocal scans, which we use as ground truth. The data set will be made available for research purposes.
Confocal and multiphoton microscopy are effective techniques to obtain high-contrast images of 2-D sections within bulk tissue. However, scattering limits their application to depths only up to ~1 millimeter. Multimode fibers make excellent ultrathin endoscopes that can penetrate deep inside the tissue with minimal damage. Here, we present Multiview Scattering Scanning Imaging Confocal (MUSSIC) Microscopy that enables high signal-to-noise ratio (SNR) imaging through a multimode fiber, hence combining the optical sectioning and resolution gain of confocal microscopy with the minimally invasive …
Confocal microscopy is essential for histopathologic cell visualization and quantification. Despite its significant role in biology, fluorescence confocal microscopy suffers from the presence of inherent noise during image acquisition. Non-local patch-wise Bayesian mean filtering (NLB) was until recently the state-of-the-art denoising approach. However, classic denoising methods have been outperformed by neural networks in recent years. In this work, we propose to exploit the strengths of NLB in the framework of Bayesian deep learning. We do so by designing a convolutional neural network and t…
Spatial resolution is one of the most important specifications of an imaging system. Recent results in quantum parameter estimation theory reveal that an arbitrarily small distance between two incoherent point sources can always be efficiently determined through the use of a spatial mode sorter. However, extending this procedure to a general object consisting of many incoherent point sources remains challenging, due to the intrinsic complexity of multi-parameter estimation problems. Here, we generalize the Richardson-Lucy (RL) deconvolution algorithm to address this challenge. We simulate its …
Hamburg University of Technology（汉堡工业大学）；University Medical Centre Schleswig-Holstein（石勒苏益格-荷尔斯泰因大学医学中心）
Purpose: The gold standard for colorectal cancer metastases detection in the peritoneum is histological evaluation of a removed tissue sample. For feedback during interventions, real-time in-vivo imaging with confocal laser microscopy has been proposed for differentiation of benign and malignant tissue by manual expert evaluation. Automatic image classification could improve the surgical workflow further by providing immediate feedback. Methods: We analyze the feasibility of classifying tissue from confocal laser microscopy in the colon and peritoneum. For this purpose, we adopt both classical…
We present a laser scanning reflection-matrix microscopy combining the scanning of laser focus and the wide-field mapping of the electric field of the backscattered waves for eliminating higher-order aberrations even in the presence of strong multiple light scattering noise. Unlike conventional confocal laser scanning microscopy, we record the amplitude and phase maps of reflected waves from the sample not only at the confocal pinhole, but also at other non-confocal points. These additional measurements lead us to constructing a time-resolved reflection matrix, with which the sample-induced ab…
Enhanced Confocal Laser Scanning Microscopy with Adaptive Physics Informed Deep Autoencoders
Georgia State University（佐治亚州立大学）；Riphah International University（里法国际大学）；Roots IVY（鲁茨艾维）；Arid Agriculture University（干旱农业大学）；University of Central Florida（中佛罗里达大学）
We consider the problem of reconstructing 2D images from randomly under-sampled confocal microscopy samples. The well known and widely celebrated total variation regularization, which is the L1 norm of derivatives, turns out to be unsuitable for this problem; it is unable to handle both noise and under-sampling together. This issue is linked with the notion of phase transition phenomenon observed in compressive sensing research, which is essentially the break-down of total variation methods, when sampling density gets lower than certain threshold. The severity of this breakdown is determined b…
Learned, uncertainty-driven adaptive acquisition for photon-efficient scanning microscopy
Fu Foundation School of Engineering and Applied Science, Columbia University in the City of New York（纽约市哥伦比亚大学傅氏工程与应用科学学院）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Department of Biological Engineering, Massachusetts Institute of Technology（麻省理工学院生物工程系）
We present a dual-stage neural network architecture for analyzing fine shape details from microscopy recordings in 3D. The system, tested on red blood cells, uses training data from both healthy donors and patients with a congenital blood disease. Characteristic shape features are revealed from the spherical harmonics spectrum of each cell and are automatically processed to create a reproducible and unbiased shape recognition and classification for diagnostic and theragnostic use.
Compressed sensing fluorescence microscopy (CS-FM) proposes a scheme whereby less measurements are collected during sensing and reconstruction is performed to recover the image. Much work has gone into optimizing the sensing and reconstruction portions separately. We propose a method of jointly optimizing both sensing and reconstruction end-to-end under a total measurement constraint, enabling learning of the optimal sensing scheme concurrently with the parameters of a neural network-based reconstruction network. We train our model on a rich dataset of confocal, two-photon, and wide-field micr…
Light scattering by tissue severely limits how deep beneath the surface one can image, and the spatial resolution one can obtain from these images. Diffuse optical tomography (DOT) is one of the most powerful techniques for imaging deep within tissue -- well beyond the conventional $\sim$10-15 mean scattering lengths tolerated by ballistic imaging techniques such as confocal and two-photon microscopy. Unfortunately, existing DOT systems are limited, achieving only centimeter-scale resolution. Furthermore, they suffer from slow acquisition times and slow reconstruction speeds making real-time i…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Northeastern University（东北大学）；Caliber Imaging and Diagnostics（Caliber影像与诊断公司）；SUNY Downstate Medical Center（纽约州立大学下州医学中心）；Skin Medical Research Diagnostics, P.L.L.C.（皮肤医学研究诊断有限责任公司）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
In-vivo optical microscopy is advancing into routine clinical practice for non-invasively guiding diagnosis and treatment of cancer and other diseases, and thus beginning to reduce the need for traditional biopsy. However, reading and analysis of the optical microscopic images are generally still qualitative, relying mainly on visual examination. Here we present an automated semantic segmentation method called "Multiscale Encoder-Decoder Network (MED-Net)" that provides pixel-wise labeling into classes of patterns in a quantitative manner. The novelty in our approach is the modeling of textura…
We propose a 3D convolutional neural network to simultaneously segment and detect cell nuclei in confocal microscopy images. Mirroring the co-dependency of these tasks, our proposed model consists of two serial components: the first part computes a segmentation of cell bodies, while the second module identifies the centers of these cells. Our model is trained end-to-end from scratch on a mouse parotid salivary gland stem cell nuclei dataset comprising 107 image stacks from three independent cell preparations, each containing several hundred individual cell nuclei in 3D. In our experiments, we …
Self-Tuning Regularization for Image Scanning Microscopy
MaLGa Center, DIBRIS, University of Genoa（MaLGa中心，DIBRIS，热那亚大学）；MMS, Istituto Italiano di Tecnologia (IIT)（MMS，意大利技术研究院（IIT））；CSML, Istituto Italiano di Tecnologia (IIT)（CSML，意大利技术研究院（IIT））
Fluorescence microscopy has enabled a dramatic development in modern biology. Due to its inherently weak signal, fluorescence microscopy is not only much noisier than photography, but also presented with Poisson-Gaussian noise where Poisson noise, or shot noise, is the dominating noise source. To get clean fluorescence microscopy images, it is highly desirable to have effective denoising algorithms and datasets that are specifically designed to denoise fluorescence microscopy images. While such algorithms exist, no such datasets are available. In this paper, we fill this gap by constructing a …
