Fourier ptychographic microscopy allows for the collection of images with a high space-bandwidth product at the cost of temporal resolution. In Fourier ptychographic microscopy, the light source of a conventional widefield microscope is replaced with a light-emitting diode (LED) matrix, and multiple images are collected with different LED illumination patterns. From these images, a higher-resolution image can be computationally reconstructed without sacrificing field-of-view. We use deep learning to achieve single-shot imaging without sacrificing the space-bandwidth product, reducing the acquisition time in Fourier ptychographic microscopy by a factor of 69. In our deep learning approach, a training dataset of high-resolution images is used to jointly optimize a single LED illumination pattern with the parameters of a reconstruction algorithm. Our work paves the way for high-throughput imaging in biological studies.
We report a compact, cost-effective and field-portable lensless imaging platform for quantitative microscopy. In this platform, the object is placed on top of an image sensor chip without using any lens. We use a low-cost galvo scanner to rapidly scan an unknown laser speckle pattern on the object. To address the positioning repeatability and accuracy issues, we directly recover the positional shifts of the speckle pattern based on the phase correlation of the captured images. To bypass the resolution limit set by the imager pixel size, we employ a sub-sampled ptychographic phase retrieval pro…
Fourier ptychographic microscopy (FPM) is a promising computational imaging technique with high resolution, wide field-of-view (FOV) and quantitative phase recovery. So far, a series of system errors that may corrupt the image quality of FPM has been reported. However, an imperceptible artifact caused by edge effect caught our attention and may also degrade the precision of phase imaging in FPM with a cross-shape artifact in the Fourier space. We found that the precision of reconstructed phase at the same subregion depends on the different sizes of block processing as a result of different edg…
Fourier ptychographic microscopy (FPM) can achieve quantitative phase imaging with a large space-bandwidth product by synthesizing a set of low-resolution intensity images captured under angularly varying illuminations. Determining accurate illumination angles is critical because the consistency between actual systematic parameters and those used in the recovery algorithm is essential for high-quality imaging. This paper presents a full-pose-parameter and physics-based method for calibrating illumination angles. Using a physics-based model constructed with general knowledge of the employed mic…
We report an angle-tilted, wavelength-multiplexed ptychographic modulation approach for multispectral lensless on-chip microscopy. In this approach, we illuminate the specimen with lights at 5 wavelengths simultaneously. A prism is added at the illumination path for spectral dispersion. Lightwaves at different wavelengths, thus, hit the specimen at slightly different incident angles, breaking the ambiguities in mixed state ptychographic reconstruction. At the detection path, we place a thin diffuser in-between the specimen and the monochromatic image sensor for encoding the spectral informatio…
We report a new coherent imaging technique, termed ptychographic structured modulation (PSM), for quantitative super-resolution microscopy. In this technique, we place a thin diffuser (i.e., a scattering lens) in between the sample and the objective lens to modulate the complex light waves from the object. The otherwise inaccessible high-resolution object information can thus be encoded into the captured images. We then employ a ptychographic phase retrieval process to jointly recover the exit wavefront of the complex object and the unknown diffuser profile. Unlike the illumination-based super…
Fourier ptychography microscopy (FPM) is a new computational imaging technique that can provide gigapixel images with both high resolution and a wide field of view (FOV). However, time consuming of the data-acquisition process is a critical issue. In this paper, we make an analysis on the FPM imaging system with half number of the captured images. Based on the image analysis of the conventional FPM system, we then compare the reconstructed images with different number of captured data. Simulation and experiment results show that the reconstructed image with half number captured data do not sho…
Birefringence, an inherent characteristic of optically anisotropic materials, is widely utilized in various imaging applications ranging from material characterizations to clinical diagnosis. Polarized light microscopy enables high-resolution, high-contrast imaging of optically anisotropic specimens, but it is associated with mechanical rotations of polarizer/analyzer and relatively complex optical designs. Here, we present a novel form of polarization-sensitive microscopy capable of birefringence imaging of transparent objects without an optical lens and any moving parts. Our method exploits …
We report a novel microscopy platform, termed Multi-Aperture Fourier ptychographic microscopy (MA-FPM), capable of realizing gigapixel complex field images with large data acquisition bandwidths - in gigapixels per second. MA-FPM is a synthetic aperture technique- an array of objectives together with tilt-shift illumination are used to synthesize high-resolution, wide field-of-view images. Here, the phase is recovered using Fourier ptychography (FP) algorithms, unlike conventional optical synthetic aperture techniques where holographic measurements are used. The parallel data-acquisition capab…
We collaborated with Amgen and spent five years in designing and fabricating next generation multi-well plate imagers based on Fourier ptychographic microscopy (FPM). A 6-well imager (Emsight) and a low-cost parallel microscopic system (96 Eyes) based on parallel FPM were reported in our previous work. However, the effect of liquid meniscus on the image quality is much stronger than anticipated, introducing obvious wavevector misalignment and additional image aberration. To this end, an adaptive wavevector correction (AWC-FPM) algorithm and a pupil recovery improvement strategy are presented t…
We report a novel lensless on-chip microscopy platform based on near-field blind ptychographic modulation. In this platform, we place a thin diffuser in between the object and the image sensor for light wave modulation. By blindly scanning the unknown diffuser to different x-y positions, we acquire a sequence of modulated intensity images for quantitative object recovery. Different from previous ptychographic implementations, we employ a unit magnification configuration with a Fresnel number of ~50,000, which is orders of magnitude higher than previous ptychographic setups. The unit magnificat…
Fourier Ptychographic Microscopy (FPM) is a computational imaging method that is able to super-resolve features beyond the diffraction-limit set by the objective lens of a traditional microscope. This is accomplished by using synthetic aperture and phase retrieval algorithms to combine many measurements captured by an LED array microscope with programmable source patterns. FPM provides simultaneous large field-of-view and high resolution imaging, but at the cost of reduced temporal resolution, thereby limiting live cell applications. In this work, we learn LED source pattern designs that compr…
Fourier ptychographic microscopy (FPM) is a computational imaging technique with both high resolution and large field-of-view. However, the effective numerical aperture (NA) achievable with a typical LED panel is ambiguous and usually relies on the repeated tests of different illumination NAs. The imaging quality of each raw image usually depends on the visual assessments, which is subjective and inaccurate especially for those dark field images. Moreover, the acquisition process is really time-consuming.In this paper, we propose a SNR-based adaptive acquisition method for quantitative evaluat…
Full-color imaging is significant in digital pathology. Compared with a grayscale image or a pseudo-color image that only contains the contrast information, it can identify and detect the target object better with color texture information. Fourier ptychographic microscopy (FPM) is a high-throughput computational imaging technique that breaks the tradeoff between high resolution (HR) and large field-of-view (FOV), which eliminates the artifacts of scanning and stitching in digital pathology and improves its imaging efficiency. However, the conventional full-color digital pathology based on FPM…
In example-based super-resolution, the function relating low-resolution images to their high-resolution counterparts is learned from a given dataset. This data-driven approach to solving the inverse problem of increasing image resolution has been implemented with deep learning algorithms. In this work, we explore modifying the imaging hardware in order to collect more informative low-resolution images for better ultimate high-resolution image reconstruction. We show that this "physical preprocessing" allows for improved image reconstruction with deep learning in Fourier ptychographic microscop…
Direct Image Classification from Fourier Ptychographic Microscopy Measurements without Reconstruction
Kalinga Institute of Industrial Technology, India（印度卡林加工业技术学院）；University of Siegen, Germany（德国施派尔大学）；Chair for Computer Vision, University of Siegen, Germany（德国施派尔大学计算机视觉研究所）；Chair for Computational Sensing and Communications Engineering, University of Siegen, Germany（德国施派尔大学计算感知与通信工程研究所）
X-ray ptychography imaging at synchrotron facilities like the Advanced Photon Source (APS) involves controlling instrument hardwares to collect a set of diffraction patterns from overlapping coherent illumination spots on extended samples, managing data storage, reconstructing ptychographic images from acquired diffraction patterns, and providing the visualization of results and feedback. In addition to the complicated workflow, ptychography instrument could produce up to several TB's of data per second that is needed to be processed in real time. This brings up the need to develop a high perf…
Ptychography is a technique widely used in microscopy for achieving high-resolution imaging. This method relies on computational processing of images gathered from diffraction patterns produced by several partial illuminations of a sample. In this work, we numerically studied the effect of using different shapes for illuminating the aforementioned sample: convex shapes, such as circles and regular polygons, and unconnected shapes that resemble a QR code. Our results suggest that the use of unconnected shapes seems to outperform convex shapes in terms of convergence and, in some cases, accuracy…
Tsinghua University（清华大学）；University of Connecticut（康涅狄格大学）
Fourier ptychography (FP) is a newly developed computational imaging approach that achieves both high resolution and wide field of view by stitching a series of low-resolution images captured under angle-varied illumination. So far, many supervised data-driven models have been applied to solve inverse imaging problems. These models need massive amounts of data to train, and are limited by the dataset characteristics. In FP problems, generic datasets are always scarce, and the optical aberration varies greatly under different acquisition conditions. To address these dilemmas, we model the forwa…
The structured illumination microscopy using unknown speckle patterns has shown the capacity to surpass the Abbe's diffraction barrier, giving the possibility to design cheap and versatile SIM devices. However, the state-of-the-art joint reconstruction methods in this framework has a relatively low contrast in super-resolution part in comparison to conventional SIM and the hyper-parameter is not easy to tune. In this paper, a unified joint reconstruction approach is proposed with the hyper-parameter proportional to the noise level. Different regularization terms could be evaluated under the sa…
Argonne National Laboratory（阿贡国家实验室）；University of Illinois（伊利诺伊大学）；NVIDIA Corporation（英伟达公司）
Coherent microscopy techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent X-ray microscopy methods like ptychography are poised to revolutionize nanoscale materials characterization. However, associated significant increases in data and compute needs mean that conventional approaches no longer suffice for recovering sample images in real-time from high-speed coherent im…
X-ray ptychography is a cutting edge imaging technique providing ultra-high spatial resolutions. In ptychography, phase retrieval, i.e., the recovery of a complex valued signal from intensity-only measurements, is enabled by exploiting a redundancy of information contained in diffraction patterns measured with overlapping illuminations. For samples that are considerably larger than the probe we show that during the iteration the bulk information has to propagate from the sample edges to the center. This constitutes an inherent limitation of reconstruction speed for algorithms that use a flat i…
Ptychography has risen as a reference X-ray imaging technique: it achieves resolutions of one billionth of a meter, macroscopic field of view, or the capability to retrieve chemical or magnetic contrast, among other features. A ptychographyic reconstruction is normally formulated as a blind phase retrieval problem, where both the image (sample) and the probe (illumination) have to be recovered from phaseless measured data. In this article we address a nonlinear least squares model for the blind ptychography problem with constraints on the image and the probe by maximum likelihood estimation of…
High-content biological microscopy targets high-resolution imaging across large fields-of-view (FOVs). Recent works have demonstrated that computational imaging can provide efficient solutions for high-content microscopy. Here, we use speckle structured illumination microscopy (SIM) as a robust and cost-effective solution for high-content fluorescence microscopy with simultaneous high-content quantitative phase (QP). This multi-modal compatibility is essential for studies requiring cross-correlative biological analysis. Our method uses laterally-translated Scotch tape to generate high-resoluti…
Nanjing University of Science and Technology（南京理工大学）；Soochow University（苏州大学）
Structured illumination microscopy (SIM) is an important super-resolution based microscopy technique that breaks the diffraction limit and enhances optical microscopy systems. With the development of biology and medical engineering, there is a high demand for real-time and robust SIM imaging under extreme low light and short exposure environments. Existing SIM techniques typically require multiple structured illumination frames to produce a high-resolution image. In this paper, we propose a single-frame structured illumination microscopy (SF-SIM) based on deep learning. Our SF-SIM only needs o…
A method is proposed for assessing the temporal resolution of Structured Illumination Microscopy (SIM), by tracking the amplitude of different spatial frequency components over time, and comparing them to a temporally-oscillating ground-truth. This method is used to gain insight into the performance limits of SIM, along with alternative reconstruction techniques (termed 'rolling SIM') that claim to improve temporal resolution. Results show that the temporal resolution of SIM varies considerably between low and high spatial frequencies, and that, despite being used in several high profile paper…
Structured illumination microscopy (SIM) can double the resolution beyond the light diffraction limit, but it comes at the cost of multiple camera exposures and the heavy computation burden of multiple Fourier transforms. In this paper, we report a novel technique termed single-shot SIM, to overcome these limitations. A multi-task joint deep-learning strategy is proposed. Generative adversative networks (GAN) are employed to generate five structured illumination images based on the single-shot structured illumination image. U-Net is employed to reconstruct the super-resolution image from these…
Structured Illumination Microscopy (SIM) is an imaging technique for achieving both super-resolution (SR) and optical sectioning (OS) in wide-field microscopy. It consists in illuminating the sample with periodic patterns at different orientations and positions. The resulting images are then processed to reconstruct the observed object with SR and/or OS. In this work, we present BOSSA-SIM, a general-purpose SIM reconstruction method, applicable to moving objects such as encountered in in vivo retinal imaging, that enables SR and OS jointly in a fully unsupervised Bayesian framework. By modelin…
Structured illumination microscopy (SIM) achieves doubled spatial resolution by exciting the specimen with a high-contrast, high-frequency sinusoidal pattern. Such an excitation pattern can be generated by interference between multiple laser beams, which are diffracted from a grating. In SIM, 2D imaging requires 9 patterns and 3D imaging requires 15 patterns. Compared to mechanical movement of gratings, opti-electro devices provide rapid switch of the excitation patterns, in which Digital Micro-mirror Device (DMD) is most common in industry. Here we model DMD as the blazed grating and report a…
Imaging of biological cells and tissues often relies on fluorescent labels, which offer high contrast with molecular specificity. The use of exogenous labeling agents, however, may alter the normal physiology of the bio-specimens. Complementary to the established fluorescence microscopy, label-free quantitative phase imaging provides an objective morphological measurement tool for bio-specimens and is free of variability introduced by contrast agents. Here we report a simple and low-cost microscope add-on, termed Ptychographic Modulation Engine (PME), for super-resolution quantitative phase im…
Traditional microbial detection methods often rely on the overall property of microbial cultures and cannot resolve individual growth event at high spatiotemporal resolution. As a result, they require bacteria to grow to confluence and then interpret the results. Here, we demonstrate the application of an integrated ptychographic sensor for lensless cytometric analysis of microbial cultures over a large scale and with high spatiotemporal resolution. The reported device can be placed within a regular incubator or used as a standalone incubating unit for long-term microbial monitoring. For longi…
In recent years there has been great interest in using deep neural networks (DNN) for super-resolution image reconstruction including for structured illumination microscopy (SIM). While these methods have shown very promising results, they all rely on data-driven, supervised training strategies that need a large number of ground truth images, which is experimentally difficult to realize. For SIM imaging, there exists a need for a flexible, general, and open-source reconstruction method that can be readily adapted to different forms of structured illumination. We demonstrate that we can combine…
Digital two-dimensional (2D) spatial sampling devices (such as charge-coupled device) have been widely used in various imaging systems, especially in computational imaging systems. However, the undersampling of digital sampling devices is a problem that limits the resolution of the acquired images. In this study, we present a synthetic sampling imaging (SSI) concept to solve the undersampling problem. It combines the structured illumination system and conventional 2D image detection system to simultaneously sample the specimen from the illumination and the detection sides. Then, we synthesize …
Many optical microscopy techniques rely on structured illumination by way of a projected image (eg. structured illumination microscopy) or a tailored angular distribution of light (eg. ptychography). Electro-optical equipment such as spatial light modulators and light emitting diode (LED) arrays are commonly used to sculpt light for these imaging schemes. However, these high-tech devices are not a requirement for light crafting. Patterned diffusely reflecting surfaces also imprint a spatio-angular structure onto reflected light. We demonstrate that paper patterned with a standard commercial pr…
The performance of light-field microscopy is improved by selectively illuminating the relevant subvolume of the specimen with a second objective lens [1-3]. Here we advance this approach to a single-objective geometry, using an oblique one-photon illumination path or two-photon illumination to accomplish selective-volume excitation. The elimination of the second orthogonally oriented objective to selectively excite the volume of interest simplifies specimen mounting; yet, this single-objective approach still reduces out-of-volume background, resulting in improvements in image contrast, effecti…
The transport-of-intensity equation (TIE) is a well-established non-interferometric phase retrieval approach, which enables quantitative phase imaging (QPI) of transparent sample simply by measuring the intensities at multiple axially displaced planes. Nevertheless, it still suffers from two fundamentally limitations. First, it is quite susceptible to low-frequency errors (such as \cloudy" artifacts), which results from the poor contrast of the phase transfer function (PTF) near the zero frequency. Second, the reconstructed phase tends to blur under spatially low-coherent illumination, especia…
This work presents the development of a flexible microscopic chemical imaging platform for cultural heritage that utilizes wavelength-tunable oblique illumination from a point source to obtain per-pixel reflectance spectra in the VIS-NIR range. The microscope light source can be adjusted on two axes allowing for a hemisphere of possible illumination directions. The synthesis of multiple illumination angles allows for the calculation of surface normal vectors, similar to phase gradients, and axial optical sectioning. The extraction of spectral reflectance images with high spatial resolutions fr…
University of Cambridge（剑桥大学）；Department of Chemical Engineering and Biotechnology（化学工程与生物技术系）；Department of Computer Science and Technology（计算机科学与技术系）
Structured illumination microscopy (SIM) has become an important technique for optical super-resolution imaging because it allows a doubling of image resolution at speeds compatible for live-cell imaging. However, the reconstruction of SIM images is often slow and prone to artefacts. Here we propose a versatile reconstruction method, ML-SIM, which makes use of machine learning. The model is an end-to-end deep residual neural network that is trained on a simulated data set to be free of common SIM artefacts. ML-SIM is thus robust to noise and irregularities in the illumination patterns of the r…
This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. Our results show that by including a model of illumination within the first layers of a deep convolutional neural network, it is possible to learn task-specific LED patterns that substantially improve the ability to infer fluorescence image information from unstained transmission microscopy images. We validated our method on two different experimental setups, with different magnifications and different sample types, to show a consistent improvement in performance as compared to conventional…
