Deep learning has become an extremely effective tool for image classification and image restoration problems. Here, we apply deep learning to microscopy, and demonstrate how neural networks can exploit the chromatic dependence of the point-spread function to classify the colors of single emitters imaged on a grayscale camera. While existing single-molecule methods for spectral classification require additional optical elements in the emission path, e.g. spectral filters, prisms, or phase masks, our neural net correctly identifies static as well as mobile emitters with high efficiency using a standard, unmodified single-channel configuration. Furthermore, we demonstrate how deep learning can be used to design phase-modulating elements that, when implemented into the imaging path, result in further improved color differentiation between species.
Multicolor super-resolution imaging remains an intractable challenge for both far-field and near-field based super-resolution techniques. Planar super-oscillatory lens (SOL), a far-field subwavelength-focusing diffractive lens device, holds great potential for achieving sub-diffraction-limit imaging at multiple wavelengths. However, conventional SOL devices suffer from a numerical aperture (NA) related intrinsic tradeoff among the depth of focus (DoF), chromatic dispersion and focus spot size, being an essential characteristics of common diffractive optical elements. Typically, the limited DoF…
Electron microscopy (EM) has been instrumental in our understanding of biological systems ranging from subcellular structures to complex organisms. Although EM reveals cellular morphology with nanoscale resolution, it does not provide information on the location of proteins within a cellular context. An EM-based bioimaging technology capable of localizing individual proteins and resolving protein-protein interactions with respect to cellular ultrastructure would provide important insights into the molecular biology of a cell. Here, we report on the development of luminescent nanoprobes potenti…
Sub-diffraction resolution, gentle sample illumination, and the possibility to image in multiple colors make Structured Illumination Microscopy (SIM) an imaging technique which is particularly well suited for live cell observations. Here, we present Machine learning Assisted Interferometric-SIM (MAI-SIM), an easy-to-implement method for high speed SIM imaging in multiple colors. The instrument is based on an interferometer design in which illumination patterns are generated, rotated, and stepped in phase through movement of a single galvanometric mirror element. The design is robust, flexible,…
Single molecule localization microscopy is a recently developed superresolution imaging technique to visualize structural properties of single cells. The basic principle consists in chemically attaching fluorescent dyes to the molecules, which after excitation with a strong laser may emit light. To achieve superresolution, signals of individual fluorophores are separated in time. In this paper we follow the physical and chemical literature and derive mathematical models describing the propagation of light emitted from dyes in single molecule localization microscopy experiments via Maxwell's eq…
Point source localization is a problem of persistent interest in optical imaging. In particular, a number of widely used biological microscopy techniques rely on precise three-dimensional localization of single fluorophores. As emitter depth localization is more challenging than lateral localization, considerable effort has been spent on engineering the response of the microscope in a way that reveals increased depth information. Here we consider the theoretical limits of such approaches by deriving the quantum Cramér-Rao bound (QCRB). We show that existing methods for depth localization with …
Structured illumination in Single Molecule Localization Microscopy provides new information on the position of molecules and thus improves the localization precision compared to standard localization methods. Here, we used a time-shifted sinusoidal excitation pattern to modulate the fluorescence signal of the molecules whose position information is carried by the phase and recovered by synchronous demodulation. We designed two flexible fast demodulation systems located upstream of the camera, allowing us to overcome the limiting camera acquisition frequency and thus to maximize the collection …
Super-resolution localization microscopy is based on determining the positions of individual fluorescent markers in a sample. The major challenge in reaching an ever higher localization precision lies in the limited number of collected photons from single emitters. To tackle this issue, it has been shown that one can exploit the increased photostability at low temperatures, reaching localization precisions in the sub-nanometer range. Another crucial ingredient of single-molecule super-resolution imaging is the ability to activate individual emitter within a diffraction-limited spot. Here, we r…
Single-molecule microscopy has become an indispensable tool for biochemical analysis. The capability of characterizing distinct properties of individual molecules without averaging has provided us with a different perspective for the existing scientific issues and phenomena. Recently, the super-resolution fluorescence microscopy techniques have overcome the optical diffraction limit by the localization of molecule positions. However, the labelling process can potentially modify the intermolecular dynamics. Based on the highly-sensitive nanomechanical photothermal microscopy reported previously…
Achieving localization with molecular precision has been of great interest for extending fluorescence microscopy to nanoscopy. MINFLUX pioneers this transition through point spread function (PSF) engineering, yet its performance is primarily limited by the signal-to-background ratio. Here we demonstrate that applying two-photon excitation to MINFLUX would double its localization precision through PSF engineering by nonlinear effect. Cramér-Rao Bound (CRB) is studied as the maximum localization precision, and CRB of two-photon MINFLUX is halved compared to single-photon MINFLUX in all three dim…
We demonstrate a new versatile method for 3D super-resolution microscopy by using a deformable mirror to shape the point spread function of our microscope in a continuous and controllable way. We apply this for 3D STORM imaging of microtubules.
We present a coherent population trapping based scheme to attain sub-nanoscale resolution for atom localization, microscopy and lithography. Our method uses three-level atoms coupled to amplitude modulated probe field and spatially dependent drive field. The modulation of the probe field allows us to tap into the steep dispersion normally associated with electromagnetically induced transparency and offers an avenue to attain sub-nanometer resolution using just optical fields. We illustrate application of the techniques to the area of microscopy and lithography and show how multilevel schemes o…
The curvature of biological membranes at the nanometer scale is critically important for vesicle trafficking, organelle morphology, and disease propagation. This manuscript reports the development of Polarized Localization Microscopy (PLM), a pointillist optical imaging technique for the detection of nanoscale membrane curvature in correlation with single-molecule dynamics and molecular sorting. PLM combines polarized total internal reflection fluorescence microscopy (TIRFM) and single-molecule localization microscopy to reveal membrane orientation with sub-diffraction-limited resolution witho…
We propose a straightforward sample-based technique to calibrate the axial detection in 3D single molecule localization microscopy (SMLM). Using microspheres coated with fluorescent molecules, the calibration curves of PSF-shaping- or intensity-based measurements can be obtained for any required depth range from a few hundreds of nanometers to several tens of microns. This experimental method takes into account the effect of the spherical aberration without requiring computational correction.
A standard paradigm of localization microscopy involves extension from two to three dimensions by engineering information into emitter images, and approximation of errors resulting from the field dependence of optical aberrations. We invert this standard paradigm, introducing the concept of fully exploiting the latent information of intrinsic aberrations by comprehensive calibration of an ordinary microscope, enabling accurate localization of single emitters in three dimensions throughout an ultrawide and deep field. To complete the extraction of spatial information from microscale bodies rang…
Nanoscale localization of point emitters is critical to several methods in optical fluorescence microscopy, including single-molecule super-resolution imaging and tracking. While the precision of the localization procedure has been the topic of extensive study, localization accuracy has been less emphasized, in part due to the challenge of producing an experimental sample containing unperturbed point emitters at known three-dimensional positions in a relevant geometry. We report a new experimental system which reproduces a widely-adopted geometry in high-numerical aperture localization microsc…
Traceability to the International System of Units (SI) is fundamental to measurement accuracy and reliability. In this study, we demonstrate subnanometer traceability of localization microscopy, establishing a metrological foundation for the maturation and application of super-resolution methods. To do so, we create a master standard by measuring the positions of submicrometer apertures in an array by traceable atomic-force microscopy. We perform correlative measurements of this master standard by optical microscopy, calibrating scale factor and correcting aberration effects. We introduce the …
Here, we report analysis and summary of research in the field of localization microscopy for optical imaging. We introduce the basic elements of super-resolved localization microscopy methods for PALM and STORM, commonly used both in vivo and in vitro, discussing the core essentials of background theory, instrumentation and computational algorithms. We discuss the resolution limit of light microscopy and the mathematical framework for localizing fluorescent dyes in space beyond this limit, including the precision obtainable as a function of the amount of light emitted from a dye, and how it le…
Background fluorescence, especially when it exhibits undesired spatial features, is a primary factor for reduced image quality in optical microscopy. Structured background is particularly detrimental when analyzing single-molecule images for 3D localization microscopy or single-molecule tracking. Here, we introduce BGnet, a deep neural network with a U-net-type architecture, as a general method to rapidly estimate the background underlying the image of a point source with excellent accuracy, even when point spread function (PSF) engineering is in use to create complex PSF shapes. We trained BG…
Interferometric imaging is an emerging technique for particle tracking and mass photometry. Mass or position are estimated from weak signals, coherently scattered from nanoparticles or single molecules, and interfered with a co-propagating reference. In this work, we perform a statistical analysis and derive lower bounds on the measurement precision of the parameters of interest from shot-noise limited images. This is done by computing the classical Cramér-Rao bound for localization and mass estimation, using a precise vectorial model of interferometric imaging techniques. We then derive funda…
FRET-based approaches are a unique tool for sensing the immediate surroundings and interactions of (bio)molecules. FRET imaging and FLIM (Fluorescence Lifetime Imaging Microscopy) enable the visualization of the spatial distribution of molecular interactions and functional states. However, conventional FLIM and FRET imaging provide average information over an ensemble of molecules within a diffraction-limited volume, which limits the spatial information, accuracy, and dynamic range of the observed signals. Here, we demonstrate an approach to obtain super-resolved FRET imaging based on single-m…
Despite advances in low-light level detection, single-photon methods such as photon correlation have rarely been used in the context of imaging. The few demonstrations, for example of sub-diffraction limited imaging utilizing quantum statistics of photons, have remained in the realm of proof-of-principle demonstrations. This is primarily due to a combination of low values of fill factors, quantum efficiencies, frame rates and signal-to-noise characteristic of most available single-photon sensitive imaging detectors. Here we describe an imaging device based on a fiber bundle coupled to single-p…
The common assumption that precision is the limit of accuracy in localization microscopy and the typical absence of comprehensive calibration of optical microscopes lead to a widespread issue - overconfidence in measurement results with nanoscale statistical uncertainties that can be invalid due to microscale systematic errors. In this article, we report a comprehensive solution to this underappreciated problem. We develop arrays of subresolution apertures into the first reference materials that enable localization errors approaching the atomic scale across a submillimeter field. We present no…
We investigate a nonlinear localization microscopy method based on Rabi oscillations of single emitters. We demonstrate the fundamental working principle of this new technique using a cryogenic far-field experiment in which subwavelength features smaller than $λ$/10 are obtained. Using Monte Carlo simulations, we show the superior localization accuracy of this method under realistic conditions and a potential for higher acquisition speed or a lower number of required photons as compared to conventional linear schemes. The method can be adapted to other emitters than molecules and allows for th…
Localization microscopy is an imaging technique in which the positions of individual nanoscale point emitters (e.g. fluorescent molecules) are determined at high precision from their images. This is the key ingredient in single/multiple-particle-tracking and several super-resolution microscopy approaches. Localization in three-dimensions (3D) can be performed by modifying the image that a point-source creates on the camera, namely, the point-spread function (PSF). The PSF is engineered using additional optical elements to vary distinctively with the depth of the point-source. However, localizi…
We report the use of a phase retrieval procedure based on maximum likelihood estimation (MLE) to produce an improved, experimentally calibrated model of a point spread function (PSF) for use in three-dimensional (3D) localization microscopy experiments. The method estimates a global pupil phase function (which includes both the PSF and system aberrations) over the full axial range from a simple calibration scan. The pupil function is used to refine the PSF model and hence enable superior localizations from experimental data. To demonstrate the utility of the procedure, we apply it to experimen…
Various techniques have been developed to measure the 2D and 3D positions and 2D and 3D orientations of fluorescent molecules with improved precision over standard epifluorescence microscopes. Due to the challenging signal-to-background ratio in typical single-molecule experiments, it is essential to choose an imaging system optimized for the specific target sample. In this work, we compare the performance of multiple state-of-the-art and commonly used methods for orientation localization microscopy against the fundamental limits of measurement precision. Our analysis reveals optimal imaging m…
We present a new technique that combines off-axis Digital Holography and Dark Field Microscopy to track 100nm gold particles diffusing in water. We show that a single hologram is sufficient to localize several particles in a thick sample with a localization accuracy independent of the particle position. From our measurements we reconstruct the trajectories of the particles and derive their 3D diffusion coefficient. Our results pave the way for quantitative studies of the motion of single nanoparticle in complex media.
Point-wise localization of individual fluorophores is a critical step in super-resolution microscopy and single particle tracking. Although the methods are limited by the accuracy in localizing individual flourophores, this point-wise accuracy has so far only been estimated by theoretical best case approximations, disregarding for example motional blur, out of focus broadening of the point spread function and time varying changes in the fluorescence background. Here, we show that pointwise localization uncertainty can be accurately estimated directly from imaging data using a Laplace approxima…
We present an adapted single-shot convolutional neural network (YOLOv2) for the real-time localization and classification of particles in optical microscopy. As compared to previous works, we focus on the real-time detection capabilities of the system to allow for manipulation of microscopic objects in large heterogeneous ensembles with the help of feedback control. The network is capable of localizing and classifying several hundreds of microscopic objects even at very low signal-to-noise ratios for images as large as 416x416 pixels with an inference time of about 10 ms. We demonstrate the re…
Tracking cells in 3D at high speed continues to attract extensive attention for many biomedical applications, such as monitoring immune cell migration and observing tumor metastasis in flowing blood vessels. Here, we propose a deep convolutional neural networks (CNNs) based method to retrieve the 3D locations of the fluorophores from a single 2D image captured by a conventional wide-field fluorescence microscope without any hardware modification. The reported method converts the challenging 3D localization from an ill-posed model-based fitting problem, especially with dense samples and low sig…
Technion - Israel Institute of Technology（以色列理工学院）；Google Research（谷歌研究院）
A long-standing challenge in multiple-particle-tracking is the accurate and precise 3D localization of individual particles at close proximity. One established approach for snapshot 3D imaging is point-spread-function (PSF) engineering, in which the PSF is modified to encode the axial information. However, engineered PSFs are challenging to localize at high densities due to lateral PSF overlaps. Here we suggest using multiple PSFs simultaneously to help overcome this challenge, and investigate the problem of engineering multiple PSFs for dense 3D localization. We implement our approach using a…
Precisely measuring the three-dimensional position and orientation of individual fluorophores is challenging due to the substantial photon shot noise in single-molecule experiments. Facing this limited photon budget, numerous techniques have been developed to encode 2D and 3D position and 2D and 3D orientation information into fluorescence images. In this work, we adapt classical and quantum estimation theory and propose a mathematical framework to derive the best possible precision for measuring the position and orientation of dipole-like emitters for any fixed imaging system. We find that it…
We present an ultra-fast, precise, parameter-free method, which we term Deep-STORM, for obtaining super-resolution images from stochastically-blinking emitters, such as fluorescent molecules used for localization microscopy. Deep-STORM uses a deep convolutional neural network that can be trained on simulated data or experimental measurements, both of which are demonstrated. The method achieves state-of-the-art resolution under challenging signal-to-noise conditions and high emitter densities, and is significantly faster than existing approaches. Additionally, no prior information on the shape …
We present a method that can simultaneously locate positions of overlapped multi-emitters at the theoretical-limit precision. We derive a set of simple equations whose solution gives the maximum likelihood estimator of multi-emitter positions. We compare the performance of our simultaneous localization analysis with the conventional single-molecule analysis for simulated images and show that our method can improve the time-resolution of superresolution microscopy an order of magnitude. In particular, we derive the information-theoretical bound on time resolution of localization-based superreso…
We propose the use of the subwavelength localization via adiabatic passage technique for fluorescence microscopy with nanoscale resolution in the far field. This technique uses a Λ-type medium coherently coupled to two laser pulses: the pump, with a node in its spatial profile, and the Stokes. The population of the Λ system is adiabatically transferred from one ground state to the other except at the node position, yielding a narrow population peak. This coherent localization allows fluorescence imaging with nanometer lateral resolution. We derive an analytical expression to asses the resoluti…
The design of single-molecule photoswitchable emitters was the first milestone toward the advent of single-molecule localization microscopy that sets a new paradigm in the field of optical imaging. Several photoswitchable emitters have been developed but they all fluoresce in the visible or far-red ranges, missing the desirable near-infrared window where biological tissues are most transparent. Moreover, photocontrol of individual emitters in the near-infrared would be highly desirable for elementary optical molecular switches or information storage elements since most communication data trans…
An interferometric fluorescent microscope and a novel theoretic image reconstruction approach were developed and used to obtain super-resolution images of live biological samples and to enable dynamic real time tracking. The tracking utilizes the information stored in the interference pattern of both the illuminating incoherent light and the emitted light. By periodically shifting the interferometer phase and a phase retrieval algorithm we obtain information that allow localization with sub-2 nm axial resolution at 5 Hz.
Localization-based imaging has revolutionized fluorescence optical microscopy and has also enabled unprecedented ultrasound images of microvascular structures in deep tissues. Herein, we introduce a new concept of localization optoacoustic tomography (LOAT) that employs rapid sequential acquisition of three-dimensional optoacoustic images from flowing absorbing particles. We show that the new method enables breaking through the spatial resolution barrier of acoustic diffraction while further enhancing the visibility of structures under limited-view tomographic conditions. Given the intrinsic s…
We report on image processing techniques and experimental procedures to determine the lattice-site positions of single atoms in an optical lattice with high reliability, even for limited acquisition time or optical resolution. Determining the positions of atoms beyond the diffraction limit relies on parametric deconvolution in close analogy to methods employed in super-resolution microscopy. We develop a deconvolution method that makes effective use of the prior knowledge of the optical transfer function, noise properties, and discreteness of the optical lattice. We show that accurate knowledg…
Single molecule localization microscopy has the potential to resolve structural details of biological samples at the nanometer length scale. However, to fully exploit the resolution it is crucial to account for the anisotropic emission characteristics of fluorescence dipole emitters. In case of slight residual defocus, localization estimates may well be biased by tens of nanometers. We show here that astigmatic imaging in combination with information about the dipole orientation allows to extract the position of the dipole emitters without localization bias and down to a precision of ~1nm, the…
Fast 3D super-resolution imaging is essential for decoding rapidly occurring biological processes. Encoding single molecules to their respective planes enable simultaneous multi-plane super-resolution volume imaging. This saves the data-acquisition time and as a consequence reduce radiation-dose that lead to photobleaching and other undesirable photochemical reactions. Detection and subsequent identification of the locus of individual molecule (both on the focal plane and off-focal planes) holds the key. Experimentally, this is achieved by accurate calibration of system PSF size and its natura…
Localization of single fluorescent molecules is key for physicochemical and biophysical measurements such as single-molecule tracking and super-resolution imaging by single-molecule localization microscopy (SMLM). Recently a series of methods have been developed in which the localization precision is enhanced by interrogating the molecular position with a sequence of spatially modulated patterns of light. Among them, the MINFLUX technique outstands for achieving a ~10-fold improvement compared to wide-field camera-based single-molecule localization, reaching ~1-2 nm localization precision at m…
We demonstrate an electro-optic wide-field method to enable fluorescence lifetime microscopy (FLIM) with high throughput and single-molecule sensitivity. Resonantly driven Pockels cells are used to efficiently gate images at 39 MHz, allowing fluorescence lifetime to be captured on standard camera sensors. Lifetime imaging of single molecules is enabled in wide-field with exposure times of less than 100 milliseconds. This capability allows combination of wide-field FLIM with single-molecule super-resolution localization microscopy. Fast single-molecule dynamics such as FRET and molecular bindin…
Multifocal microscopy affords fast acquisition of microscopic 3D images. This is made possible using a multifocal grating optic, however this induces chromatic dispersion effects into the point spread function impacting image quality and single-molecule localization precision. To minimize this effect, researchers use narrow-band emission filters. However, the choice of optimal emission filter bandwidth in such systems is, thus far, unclear. This work presents a theoretical framework to investigate how the localization precision of a point emitter is affected by the emission filter bandwidth. W…
More than twenty years ago, scientists succeeded in pushing the limits of optical detection to single molecules using fluorescence. This breakthrough has revolutionized biophysical measurements, but restrictions in photophysics and labeling protocols have motivated many efforts to achieve fluorescence-free single-molecule sensitivity in biological studies. Although several interesting mechanisms using vibrational spectroscopy, photothermal detection, plasmonics or microcavities have been proposed for biosensing at the single-protein level, no method has succeeded in direct label-free detection…
At cryogenic temperature and at the single emitter level, the optical properties of single-wall carbon nanotubes depart drastically from that of a one-dimensional (1D) object. In fact, the (usually unintentional) localization of excitons in local potential wells leads to nearly 0D behaviors such as photon antibunching, spectral diffusion, inhomogeneous broadening, etc. Here, we present an hyperspectral imaging of this exciton self-localization effect at the single nanotube level using a super-resolved optical microscopy approach. We report on the statistical distribution of the traps localizat…
Institute of Micro- and Nanotechnologies, Technische Universität Ilmenau（微纳技术研究所，埃尔朗根-纽伦堡工业大学）；Institute of Thermodynamics and Fluid Mechanics, Technische Universität Ilmenau（热力学与流体力学研究所，埃尔朗根-纽伦堡工业大学）
