High-dynamic-range (HDR) imaging is crucial for many computer graphics and vision applications. Yet, acquiring HDR images with a single shot remains a challenging problem. Whereas modern deep learning approaches are successful at hallucinating plausible HDR content from a single low-dynamic-range (LDR) image, saturated scene details often cannot be faithfully recovered. Inspired by recent deep optical imaging approaches, we interpret this problem as jointly training an optical encoder and electronic decoder where the encoder is parameterized by the point spread function (PSF) of the lens, the bottleneck is the sensor with a limited dynamic range, and the decoder is a convolutional neural network (CNN). The lens surface is then jointly optimized with the CNN in a training phase; we fabricate this optimized optical element and attach it as a hardware add-on to a conventional camera during inference. In extensive simulations and with a physical prototype, we demonstrate that this end-to-end deep optical imaging approach to single-shot HDR imaging outperforms both purely CNN-based approaches and other PSF engineering approaches.
This paper proposes a novel multi-exposure image fusion (MEF) scheme for single-shot high dynamic range imaging with spatially varying exposures (SVE). Single-shot imaging with SVE enables us not only to produce images without color saturation regions from a single-shot image, but also to avoid ghost artifacts in the producing ones. However, the number of exposures is generally limited to two, and moreover it is difficult to decide the optimum exposure values before the photographing. In the proposed scheme, a scene segmentation method is applied to input multi-exposure images, and then the lu…
We proposes a novel single-shot high dynamic range imaging scheme with spatially varying exposures (SVE) considering hue distortion. Single-shot imaging with SVE enables us to capture multi-exposure images from a single-shot image, so high dynamic range images can be produced without ghost artifacts. However, SVE images have some pixels at which a range supported by camera sensors is exceeded. Therefore, generated images have some color distortion, so that conventional imaging with SVE has never considered the influence of this range limitation. To overcome this issue, we consider estimating t…
Imaging depth and spectrum have been extensively studied in isolation from each other for decades. Recently, hyperspectral-depth (HS-D) imaging emerges to capture both information simultaneously by combining two different imaging systems; one for depth, the other for spectrum. While being accurate, this combinational approach induces increased form factor, cost, capture time, and alignment/registration problems. In this work, departing from the combinational principle, we propose a compact single-shot monocular HS-D imaging method. Our method uses a diffractive optical element (DOE), the point…
Zero-Shot Solving of Imaging Inverse Problems via Noise-Refined Likelihood Guided Diffusion Models
School of Mathematics and Statistics（数学与统计学学院）；Nanjing University of Science and Technology（南京理工大学）；School of Computer Science and Engineering（计算机科学与工程学院）
Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research（资讯通信研究院）
Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying …
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；McMaster University（麦克马斯特大学）；Shanghai Jiao Tong University（上海交通大学）
Whole slide imaging (WSI) is an emerging technology for digital pathology. The process of autofocusing is the main influence of the performance of WSI. Traditional autofocusing methods either are time-consuming due to repetitive mechanical motions, or require additional hardware and thus are not compatible to current WSI systems. In this paper, we propose the concept of \textit{virtual autofocusing}, which does not rely on mechanical adjustment to conduct refocusing but instead recovers in-focus images in an offline learning-based manner. With the initial focal position, we only perform two-sh…
We present a method for single-shot three-dimensional imaging through scattering media with a three-dimensional memory effect. In the proposed computational process, a captured speckle image is two-dimensionally correlated with different scales, and the object is three-dimensionally recovered with three-dimensional phase retrieval. Our method was experimentally demonstrated with a lensless setup and was compared with a multi-shot approach used in our previous work [Y. Okamoto, et al., Opt. Lett. 44, 2526-2529 (2019)].
Using Few-Shot Learning to Classify Primary Lung Cancer and Other Malignancy with Lung Metastasis in Cytological Imaging via Endobronchial Ultrasound Procedures
National Yang Ming Chiao Tung University（阳明交通大学）；National Taiwan University Cancer Center（台湾大学医学院附设癌医中心医院）；National Taiwan University Hospital（台湾大学医学院附设医院）；National Taiwan University Hsin-Chu Hospital（台湾大学医学院附设新竹医院）
Conventional approaches for 3D imaging in or through scattering media are usually limited to 2D reconstruction of objects at some discontinuous locations, although the time-consuming iteration, guide-star, or complex system are implemented. How to quickly visualize dynamic 3D objects behind scattering media is still an open issue. Here, by using structured light illumination, we propose a single-shot technique that can quickly acquire continuous 3D surfaces of objects hidden behind the diffuser. The proposed method can realize the 3D imaging of single, multiple, and dynamic targets from the sp…
Based on point spread function (PSF) engineering and astigmatism due to a pair of cylindrical lenses, a novel compressed imaging mechanism is proposed to achieve single-shot incoherent 3D imaging. The speckle-like PSF of the imaging system is sensitive to axial shift, which makes it feasible to reconstruct a 3D image by solving an optimization problem with sparsity constraint. With the experimentally calibrated PSFs, the proposed method is demonstrated by a synthetic 3D point object and real 3D object, and the images in different axial slices can be reconstructed faithfully. Moreover, 3D multi…
Deep learning models perform best when tested on target (test) data domains whose distribution is similar to the set of source (train) domains. However, model generalization can be hindered when there is significant difference in the underlying statistics between the target and source domains. In this work, we adapt a domain generalization method based on a model-agnostic meta-learning framework to biomedical imaging. The method learns a domain-agnostic feature representation to improve generalization of models to the unseen test distribution. The method can be used for any imaging task, as it…
Accessing the point-spread function (PSF) of a complex optical system is important for a variety of imaging applications. However, placing an invasive point source is often impractical, and estimating it blindly with multiple frames is slow and requires a complex non-linear optimization. Here, we introduce a simple single-shot method to non-invasively recover the accurate PSF of an isoplanatic imaging system, in the context of multiple light scattering. Our approach is based on the reconstruction of any unknown sparse hidden object using the autocorrelation imaging technique, followed by a dec…
Multispectral imaging plays an important role in many applications from astronomical imaging, earth observation to biomedical imaging. However, the current technologies are complex with multiple alignment-sensitive components, predetermined spatial and spectral parameters by manufactures. Here, we demonstrate a single-shot multispectral imaging technique that gives flexibility to end-users with a very simple optical setup, thank to spatial correlation and spectral decorrelation of speckle patterns. These seemingly random speckle patterns are point spreading functions (PSFs) generated by light …
Magnetic resonance imaging serves as an essential tool for clinical diagnosis. However, it suffers from a long acquisition time. The utilization of deep learning, especially the deep generative models, offers aggressive acceleration and better reconstruction in magnetic resonance imaging. Nevertheless, learning the data distribution as prior knowledge and reconstructing the image from limited data remains challenging. In this work, we propose a novel Hankel-k-space generative model (HKGM), which can generate samples from a training set of as little as one k-space data. At the prior learning st…
Fluorescence imaging is indispensable to biology and neuroscience. The need for large-scale imaging in freely behaving animals has further driven the development in miniaturized microscopes (miniscopes). However, conventional microscopes / miniscopes are inherently constrained by their limited space-bandwidth-product, shallow depth-of-field, and the inability to resolve 3D distributed emitters. Here, we present a Computational Miniature Mesoscope (CM$^2$) that overcomes these bottlenecks and enables single-shot 3D imaging across an 8 $\times$ 7-mm$^2$ field-of-view and 2.5-mm depth-of-field, a…
Optical sectioning technology has been widely used in various fluorescence microscopes owing to its background removing capability. Here, a virtual HiLo based on edge detection (V-HiLo-ED) is proposed to achieve wide-field optical sectioning, which requires only single wide-field image. Compared with conventional optical sectioning technologies, its imaging speed can be increased by at least twice, meanwhile maintaining nice optical sectioning performance, low cost, and excellent artifact suppression capabilities. Furthermore, the new V-HiLo-ED can also be extended to other non-fluorescence im…
Towards Large-scale Single-shot Millimeter-wave Imaging for Low-cost Security Inspection
Beijing Institute of Technology（北京理工大学）；Yanotze Delta Region Academy of Beijing Institute of Technology (Jiaxing)（北京理工大学长三角研究院（嘉兴））；Tangshan Research Institute of Beijing Institute of Technology（北京理工大学唐山研究院）
T2 weighted imaging of the spine is commonly performed using fast spin echo (FSE/TSE) based sequences, resulting in long scan times and vulnerability to motion artifacts. While single shot fast spin echo sequences have been attempted, their adoption has been limited by poor spatial resolution and specific absorption rate (SAR) limitations. We investigate the use of a half-Fourier acquisition single-shot turbo spin-echo variable flip angle (HASTE-VFA) sequence for T2 weighted spine imaging. A variable refocusing flip angle echo train was first optimized for the spine to improve the point spread…
Multi-shot echo planar imaging (msEPI) is a promising approach to achieve high in-plane resolution with high sampling efficiency and low T2* blurring. However, due to the geometric distortion, shot-to-shot phase variations and potential subject motion, msEPI continues to be a challenge in MRI. In this work, we introduce acquisition and reconstruction strategies for robust, high-quality msEPI without phase navigators. We propose Blip Up-Down Acquisition (BUDA) using interleaved blip-up and -down phase encoding, and incorporate B0 forward-modeling into Hankel structured low-rank model to enable …
We introduce wave encoded acquisition and reconstruction techniques for highly accelerated echo planar imaging (EPI) with reduced g-factor penalty and image artifacts. Wave-EPI involves playing sinusoidal gradients during the EPI readout while employing interslice shifts as in blipped-CAIPI acquisitions. This spreads the aliasing in all spatial directions, thereby taking better advantage of 3D coil sensitivity profiles. The amount of voxel spreading that can be achieved by the wave gradients during the short EPI readout period is constrained by the slew rate of the gradient coils and periphera…
Bilkent University（比尔肯特大学）；National Magnetic Resonance Research Center, Bilkent University（比尔肯特大学国家磁共振研究中心）
Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here …
White light phase-shifting interference microscopy (WL-PSIM) is a prominent technique for high-resolution quantitative phase imaging (QPI) of industrial and biological specimens. However, multiple interferograms with accurate phase-shifts are essentially required in WL-PSIM for measuring the accurate phase of the object. Here, we present single-shot phase-shifting interferometric techniques for accurate phase measurement using filtered white light phase-shifting interference microscopy (F-WL-PSIM) and deep neural network (DNN). The methods are incorporated by training the DNN to generate 1) fo…
Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentation
Hong Kong Baptist University（香港浸会大学）；Narayana Superspeciality Hospital（纳拉亚纳超级专科医院）；National Institute of Technology Durgapur（杜尔加布尔国家理工学院）；Indian Institute of Technology（印度理工学院）
Purpose: To introduce a fast and improved direct reconstruction method for multi-shot diffusion weighted (msDW) scans for high-resolution studies. Methods:Multi-shot EPI methods can enable higher spatial resolution for diffusion MRI studies. Traditionally, such acquisitions required specialized reconstructions involving phase compensation to correct for inter-shot motion artifacts. The recently proposed MUSSELS reconstruction belongs to a new class of parallel imaging-based methods that recover artifact-free DWIs from msDW data without needing phase compensation. However, computational demands…
Harmonization Across Imaging Locations(HAIL): One-Shot Learning for Brain MRI
Children’s National Hospital（美国儿童国家医疗中心）；George Washington University（乔治·华盛顿大学）；Children’s Hospital Colorado（科罗拉多儿童医院）；Children’s Hospital of Philadelphia（费城儿童医院）
Depth imaging is vital for many emerging technologies with applications in augmented reality, robotics, gesture detection, and facial recognition. These applications, however, demand compact and low-power systems beyond the capabilities of state-of-the-art depth cameras. Here, we leverage ultrathin dielectric metasurfaces to demonstrate a solution that, with a single surface, replicates the functionality of a high-performance depth camera typically comprising a spatial light modulator, polarizer, and three lenses. Using cylindrical nano-scatterers that can arbitrarily modify the phase of an in…
Multi-spectral quantitative phase imaging (MS-QPI) is a cutting-edge label-free technique to determine the morphological changes, refractive index variations and spectroscopic information of the specimens. The bottleneck to implement this technique to extract quantitative information, is the need of more than two measurements for generating MS-QPI images. We propose a single-shot MS-QPI technique using highly spatially sensitive digital holographic microscope assisted with deep neural network (DNN). Our method first acquires the interferometric datasets corresponding to multiple wavelengths (λ…
Purpose: To develop an accelerated Cartesian MRF implementation using a multi-shot EPI sequence for rapid simultaneous quantification of T1 and T2 parameters. Methods: The proposed Cartesian MRF method involved the acquisition of highly subsampled MR images using a 16-shot EPI readout. A linearly varying flip angle train was used for rapid, simultaneous T1 and T2 quantification. The accuracy of parametric map estimations were improved by using an iterative projection algorithm. The results were compared to a conventional spiral MRF implementation. The acquisition time per slice was 8s and this…
We propose a novel approach and algorithm based on two preliminary tests of the optical system elements to enhance the super-resolved complex-valued imaging. The approach is developed for inverse phase imaging in a single-shot lensless optical setup. Imaging is based on wavefront modulation by a single binary phase mask. The preliminary tests compensate errors in the optical system and correct a carrying wavefront, reducing the gap between real-life experiments and computational modeling, which improve imaging significantly both qualitatively and quantitatively. These two tests are performed f…
In recent years nanoscale coherent imaging has emerged as an indispensable imaging modality allowing to surpass the resolution limit given by classical imaging optics. At the same time, attosecond science has experienced enormous progress and has revealed the ultrafast dynamics in atoms, molecules, and complex materials. Combining attosecond temporal resolution of pump-probe experiments with nanometer spatial resolution would allow studying ultrafast dynamics on the smallest spatio-temporal scales but has not been demonstrated yet. Unfortunately, the large bandwidth of attosecond pulses usuall…
Improved Multi-Shot Diffusion-Weighted MRI with Zero-Shot Self-Supervised Learning Reconstruction
Athinoula A. Martinos Center for Biomedical Imaging（阿瑟诺拉·A·马蒂诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；University of São Paulo（圣保罗大学）；Technische Universität München（慕尼黑工业大学）；Harvard-MIT Health Sciences and Technology（哈佛-麻省理工健康科学与技术部）
Lensless in-line holography is a simple, portable, and cost-effective method of imaging especially for the biomedical microscopy applications. We propose a multiplicative gradient descent optimization based method to obtain multi-depth imaging from a single hologram acquired in this imaging system. We further extend the method to achieve phase imaging from a single hologram. Negative-log-likelihood functional with the assumption of poisson noise has been used as the cost function to be minimized. The ill-posed nature of the problem is handled by the sparse regularization and the upper-bound co…
Purpose: To develop new encoding and reconstruction techniques for fast multi-contrast quantitative imaging. Methods: The recently proposed Echo Planar Time-resolved Imaging (EPTI) technique can achieve fast distortion- and blurring-free multi-contrast quantitative imaging. In this work, a subspace reconstruction framework is developed to improve the reconstruction accuracy of EPTI at high encoding accelerations. The number of unknowns in the reconstruction is significantly reduced by modeling the temporal signal evolutions using low-rank subspace. As part of the proposed reconstruction approa…
Purpose: This work aims to develop a novel distortion-free 3D-EPI acquisition and image reconstruction technique for fast and robust, high-resolution, whole-brain imaging as well as quantitative T2* mapping. Methods: 3D-Blip-Up and -Down Acquisition (3D-BUDA) sequence is designed for both single- and multi-echo 3D GRE-EPI imaging using multiple shots with blip-up and -down readouts to encode B0 field map information. Complementary k-space coverage is achieved using controlled aliasing in parallel imaging (CAIPI) sampling across the shots. For image reconstruction, an iterative hard-thresholdin…
We demonstrate single-shot compressive three-dimensional (3D) $(x, y, z)$ imaging based on interference coding. The depth dimension of the object is encoded into the interferometric spectra of the light field, resulting a $(x, y, λ)$ datacube which is subsequently measured by a single-shot spectrometer. By implementing a compression ratio up to $400$, we are able to reconstruct $1G$ voxels from a 2D measurement. Both an optimization based compressive sensing algorithm and a deep learning network are developed for 3D reconstruction from a single 2D coded measurement. Due to the fast acquisition…
Athinoula A. Martinos Center for Biomedical Imaging（阿西努拉·A·马蒂诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；Harvard-MIT Health Sciences and Technology（哈佛-麻省理工健康科学与技术项目）；MIT（麻省理工学院）；National Nanotechnology Center, National Science and Technology Development Agency（国家科学技术发展局国家纳米技术中心）
Purpose: To introduce a combined machine learning (ML) and physics-based image reconstruction framework that enables navigator-free, highly accelerated multishot echo planar imaging (msEPI), and demonstrate its application in high-resolution structural and diffusion imaging. Methods: Singleshot EPI is an efficient encoding technique, but does not lend itself well to high-resolution imaging due to severe distortion artifacts and blurring. While msEPI can mitigate these artifacts, high-quality msEPI has been elusive because of phase mismatch arising from shot-to-shot variations which preclude th…
In this chapter, we will give a brief overview of fundus photography, SLO, and OCT while discussing photoacoustic imaging potential as the next major ocular imaging modality.
