Coded aperture is a promising approach for capturing the 4-D light field (LF), in which the 4-D data are compressively modulated into 2-D coded measurements that are further decoded by reconstruction algorithms. The bottleneck lies in the reconstruction algorithms, resulting in rather limited reconstruction quality. To tackle this challenge, we propose a novel learning-based framework for the reconstruction of high-quality LFs from acquisitions via learned coded apertures. The proposed method incorporates the measurement observation into the deep learning framework elegantly to avoid relying entirely on data-driven priors for LF reconstruction. Specifically, we first formulate the compressive LF reconstruction as an inverse problem with an implicit regularization term. Then, we construct the regularization term with an efficient deep spatial-angular convolutional sub-network to comprehensively explore the signal distribution free from the limited representation ability and inefficiency of deterministic mathematical modeling. Experimental results show that the reconstructed LFs not only achieve much higher PSNR/SSIM but also preserve the LF parallax structure better, compared with state-of-the-art methods on both real and synthetic LF benchmarks. In addition, experiments show that our method is efficient and robust to noise, which is an essential advantage for a real camera system. The code is publicly available at \url{https://github.com/angmt2008/LFCA}
Coded aperture snapshot spectral imager (CASSI) is a computational imaging system that acquires a three dimensional (3D) spectral data cube by single or a few two dimensional (2D) measurements. Binary random coded apertures with square pixels are primarily implemented in CASSI systems to modulate the spectral images in spatial domain. The design and optimization of coded apertures was shown to improve the imaging performance of these systems significantly. This work proposes a different approach to code design. Instead of traditional squared tiled coded elements, hexagonal tiled elements are u…
Universidad Industrial de Santander（桑坦德工业大学）；Stanford University（斯坦福大学）
Compressive imaging using coded apertures (CA) is a powerful technique that can be used to recover depth, light fields, hyperspectral images and other quantities from a single snapshot. The performance of compressive imaging systems based on CAs mostly depends on two factors: the properties of the mask's attenuation pattern, that we refer to as "codification" and the computational techniques used to recover the quantity of interest from the coded snapshot. In this work, we introduce the idea of using time-varying CAs synchronized with spatially varying pixel shutters. We divide the exposure of…
Conventional stereoscopic displays suffer from vergence-accommodation conflict and cause visual fatigue. Integral-imaging-based displays resolve the problem by directly projecting the sub-aperture views of a light field into the eyes using a microlens array or a similar structure. However, such displays have an inherent trade-off between angular and spatial resolutions. In this paper, we propose a novel coded time-division multiplexing technique that projects encoded sub-aperture views to the eyes of a viewer with correct cues for vergence-accommodation reflex. Given sparse light field sub-ape…
This paper aims at developing a clustering approach with spectral images directly from the compressive measurements of coded aperture snapshot spectral imager (CASSI). Assuming that compressed measurements often lie approximately in low dimensional subspaces corresponding to multiple classes, state of the art methods generally obtains optimal solution for each step separately but cannot guarantee that it will achieve the globally optimal clustering results. In this paper, a low-rank subspace representation (LRSR) algorithm is proposed to perform clustering on the compressed measurements. In ad…
Key Laboratory of Photoelectronic Imaging Technology and System of Ministry of Education of China, School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院光电成像技术与系统教育部重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（特拉华大学电气与计算机工程系）
Hyperspectral image classification (HIC) is an active research topic in remote sensing. Hyperspectral images typically generate large data cubes posing big challenges in data acquisition, storage, transmission and processing. To overcome these limitations, this paper develops a novel deep learning HIC approach based on compressive measurements of coded-aperture snapshot spectral imagers (CASSI), without reconstructing the complete hyperspectral data cube. A new kind of deep learning strategy, namely 3D coded convolutional neural network (3D-CCNN) is proposed to efficiently solve for the classi…
To provide optimal depth resolution with a coded-aperture Laue diffraction microscope, an accurate position of the coded-aperture and its scanning geometry need to be known. However, finding the geometry by trial and error is a time-consuming and often challenging process because of the large number of parameters involved. In this paper, we propose an optimization approach to automate the focusing process after data is collected. We demonstrate the robustness and efficiency of the proposed approach with experimental data taken at a synchrotron facility.
The recently introduced Spatial Spectral Compressive Spectral Imager (SSCSI) has been proposed as an alternative to carry out spatial and spectral coding using a binary on-off coded aperture. In SSCSI, the pixel pitch size of the coded aperture, as well as its location with respect to the detector array, play a critical role in the quality of image reconstruction. In this paper, a rigorous discretization model for this architecture is developed, based on a light propagation analysis across the imager. The attainable spatial and spectral resolution, and the various parameters affecting them, is…
In this paper, we present a method for single-shot blind deconvolution incorporating a coded aperture (CA). In this method, we utilize the CA, inserted on the pupil plane, as support constraints in blind deconvolution. Not only an object but also a point spread function of turbulence are estimated from a single captured image by a reconstruction algorithm with the CA support. The proposed method is demonstrated by a simulation and an experiment in which point sources are recovered under severe turbulence.
Several coded exposure techniques have been proposed for acquiring high frame rate videos at low bandwidth. Most recently, a Coded-2-Bucket camera has been proposed that can acquire two compressed measurements in a single exposure, unlike previously proposed coded exposure techniques, which can acquire only a single measurement. Although two measurements are better than one for an effective video recovery, we are yet unaware of the clear advantage of two measurements, either quantitatively or qualitatively. Here, we propose a unified learning-based framework to make such a qualitative and quan…
We characterize the fundamental limits of coded aperture imaging systems up to universal constants by drawing upon a theorem of Nazarov regarding Fourier transforms. Our work is performed under a simple propagation and sensor model that accounts for thermal and shot noise, scene correlation, and exposure time. Focusing on mean square error as a measure of linear reconstruction quality, we show that appropriate application of a theorem of Nazarov leads to essentially optimal coded apertures, up to a constant multiplicative factor in exposure time. Additionally, we develop a heuristically effici…
We develop novel compressive coded rotating mirror (CCRM) camera to capture events at high frame rates in passive mode with a compact instrument design at the fraction of the cost compared to other high-speed imaging cameras. Operation of CCRM camera is based on the amplitude optical encoding (grey scale) and a continuous frame sweep across a low-cost detector using a motorized rotating mirror system which can achieve single pixel shift between adjacent frames. Amplitude encoding and continuous frame overlapping enable the CCRM camera to achieve high number of captured frames and high temporal…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；Peking University（北京大学）
Hyperspectral imaging plays a pivotal role in a wide range of applications, like remote sensing, medicine, and cytology. By acquiring 3D hyperspectral images (HSIs) via 2D sensors, the coded aperture snapshot spectral imaging (CASSI) has achieved great success due to its hardware-friendly implementation and fast imaging speed. However, for some less spectrally sparse scenes, single snapshot and unreasonable coded aperture design tend to make HSI recovery more ill-posed and yield poor spatial and spectral fidelity. In this paper, we propose a novel Progressive Content-Aware CASSI framework, dub…
We propose a novel concept of differential coded aperture snapshot spectral imaging (D-CASSI) technique exploiting the benefits of using {-1,+1} random mask, which is demonstrated by a broadband single-snapshot hyperspectral camera using compressed sensing. To double the information, we encode the image by two complementary random masks, which proved to be superior to two independent patterns. We utilize dispersed and non-dispersed encoded images captured in parallel onto a single detector. We explored several different approaches to processing the measured data, which demonstrates significant…
We report a novel generalized optical measurement system and computational approach to determine and correct aberrations in optical systems. We developed a computational imaging method capable of reconstructing an optical system's aberration by harnessing Fourier ptychography (FP) with no spatial coherence requirement. It can then recover the high resolution image latent in the aberrated image via deconvolution. Deconvolution is made robust to noise by using coded apertures to capture images. We term this method: coded aperture-based correction of aberration obtained from Fourier ptychography …
Tsinghua University（清华大学）；Massachusetts Institute of Technology（麻省理工学院）；Bell Labs（贝尔实验室）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
High resolution images are widely used in our daily life, whereas high-speed video capture is challenging due to the low frame rate of cameras working at the high resolution mode. Digging deeper, the main bottleneck lies in the low throughput of existing imaging systems. Towards this end, snapshot compressive imaging (SCI) was proposed as a promising solution to improve the throughput of imaging systems by compressive sampling and computational reconstruction. During acquisition, multiple high-speed images are encoded and collapsed to a single measurement. After this, algorithms are employed t…
EDoF-NeRF: extended depth-of-field neural radiance fields using a coded aperture camera
Department of Information Physics and Computing, Graduate School of Information Science and Technology, The University of Tokyo（信息物理与计算系，信息科学与技术研究生学校，东京大学）
We introduce and analyze the concept of space-spectrum uncertainty for certain commonly-used designs for spectrally programmable cameras. Our key finding states that, it is impossible to simultaneously capture high-resolution spatial images while programming the spectrum at high resolution. This phenomenon arises due to a Fourier relationship between the aperture used for obtaining spectrum and its corresponding diffraction blur in the (spatial) image. We show that the product of spatial and spectral standard deviations is lower bounded by λ/4π{ν_0} femto square-meters, where {ν_0} is the dens…
In the compressive spectral imaging (CSI) framework, different architectures have been proposed to recover high-resolution spectral images from compressive measurements. Since CSI architectures compactly capture the relevant information of the spectral image, various methods that extract classification features from compressive samples have been recently proposed. However, these techniques require a feature extraction procedure that reorders measurements using the information embedded in the coded aperture patterns. In this paper, a method that fuses features directly from dual-resolution comp…
Tsinghua University（清华大学）；Research Institute of Tsinghua University in Shenzhen（清华大学深圳研究院）；Boyan Technology DBA RayShaper China（博研科技（商业名称为RayShaper中国））
Consistent in-focus input imagery is an essential precondition for machine vision systems to perceive the dynamic environment. A defocus blur severely degrades the performance of vision systems. To tackle this problem, we propose a deep-learning-based framework estimating the kernel scale and orientation of the defocus blur to adjust lens focus rapidly. Our pipeline utilizes 3D ConvNet for a variable number of input hypotheses to select the optimal slice from the input stack. We use random shuffle and Gumbel-softmax to improve network performance. We also propose to generate synthetic defocuse…
We demonstrate single-shot compressive three-dimensional (3D) $(x, y, z)$ imaging based on interference coding. The depth dimension of the object is encoded into the interferometric spectra of the light field, resulting a $(x, y, λ)$ datacube which is subsequently measured by a single-shot spectrometer. By implementing a compression ratio up to $400$, we are able to reconstruct $1G$ voxels from a 2D measurement. Both an optimization based compressive sensing algorithm and a deep learning network are developed for 3D reconstruction from a single 2D coded measurement. Due to the fast acquisition…
Compressive spectral imaging enables to reconstruct the entire three-dimensional (3D) spectral cube from a few multiplexed images. Here, we develop a novel compressive spectral imaging technique using diffractive lenses. Our technique uses a coded aperture to spatially modulate the optical field from the scene and a diffractive lens such as a photon-sieve for dispersion. The coded field is passed through the diffractive lens and then measured at a few planes using a monochrome detector. The 3D spectral cube is then reconstructed from these highly compressed measurements through sparse recovery…
Adaptive block-based compressive sensing (ABCS) algorithms are studied in the context of the practical realization of compressive sensing on resource-constrained image and video sensing platforms that use single-pixel cameras, multi-pixel cameras or focal plane processing sensors. In this paper, we introduce two novel ABCS algorithms that are suitable for compressively sensing images or intra-coded video frames. Both use deterministic 2D-DCT dictionaries when sensing the images instead of random dictionaries. The first uses a low number of compressive measurements to compute the block boundary…
Tongji University（同济大学）；Universidad Industrial de Santander（桑坦德工业大学）；Kunming University of Science and Technology（昆明理工大学）；University of Delaware（特拉华大学）
This paper aims at developing a clustering approach with spectral images directly from CASSI compressive measurements. The proposed clustering method first assumes that compressed measurements lie in the union of multiple low-dimensional subspaces. Therefore, sparse subspace clustering (SSC) is an unsupervised method that assigns compressed measurements to their respective subspaces. In addition, a 3D spatial regularizer is added into the SSC problem, thus taking full advantages of the spatial information contained in spectral images. The performance of the proposed spectral image clustering a…
RIKEN（理化学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Tokyo（东京大学）；Nokia Bell Labs（诺基亚贝尔实验室）
Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) using a single coded two-dimensional (2D) measurement, in which algorithms are used to perform the inverse problem. Due to the ill-posed nature, various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. Unfortunately, the accuracy and computational complexity are unsatisfied. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurement, in thi…
Compressive spectral imaging (CSI) has emerged as an attractive compression and sensing technique, primarily to sense spectral regions where traditional systems result in highly costly such as in the near-infrared spectrum. Recently, it has been shown that spectral classification can be performed directly in the compressive domain, considering the amount of spectral information embedded in the measurements, skipping the reconstruction step. Consequently, the classification quality directly depends on the set of coding patterns employed in the sensing step. Therefore, this work proposes an end-…
Conventional stereo suffers from a fundamental trade-off between imaging volume and signal-to-noise ratio (SNR) -- due to the conflicting impact of aperture size on both these variables. Inspired by the extended depth of field cameras, we propose a novel end-to-end learning-based technique to overcome this limitation, by introducing a phase mask at the aperture plane of the cameras in a stereo imaging system. The phase mask creates a depth-dependent point spread function, allowing us to recover sharp image texture and stereo correspondence over a significantly extended depth of field (EDOF) th…
Nanjing University of Information Science and Technology（南京信息工程大学）；National University of Singapore（新加坡国立大学）
Hyperspectral compressive imaging takes advantage of compressive sensing theory to achieve coded aperture snapshot measurement without temporal scanning, and the entire three-dimensional spatial-spectral data is captured by a two-dimensional projection during a single integration period. Its core issue is how to reconstruct the underlying hyperspectral image using compressive sensing reconstruction algorithms. Due to the diversity in the spectral response characteristics and wavelength range of different spectral imaging devices, previous works are often inadequate to capture complex spectral …
To capture high-speed videos using a two-dimensional detector, video snapshot compressive imaging (SCI) is a promising system, where the video frames are coded by different masks and then compressed to a snapshot measurement. Following this, efficient algorithms are desired to reconstruct the high-speed frames, where the state-of-the-art results are achieved by deep learning networks. However, these networks are usually trained for specific small-scale masks and often have high demands of training time and GPU memory, which are hence {\bf \em not flexible} to $i$) a new mask with the same size…
In coded aperture snapshot spectral imaging (CASSI) system, the real-world hyperspectral image (HSI) can be reconstructed from the captured compressive image in a snapshot. Model-based HSI reconstruction methods employed hand-crafted priors to solve the reconstruction problem, but most of which achieved limited success due to the poor representation capability of these hand-crafted priors. Deep learning based methods learning the mappings between the compressive images and the HSIs directly achieved much better results. Yet, it is nontrivial to design a powerful deep network heuristically for …
Tsinghua University（清华大学）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）；University of Würzburg（维尔茨堡大学）
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradati…
Every day around the world, interminable terabytes of data are being captured for surveillance purposes. A typical 1-2MP CCTV camera generates around 7-12GB of data per day. Frame-by-frame processing of such enormous amount of data requires hefty computational resources. In recent years, compressive sensing approaches have shown impressive results in signal processing by reducing the sampling bandwidth. Different sampling mechanisms were developed to incorporate compressive sensing in image and video acquisition. Pixel-wise coded exposure is one among the promising sensing paradigms for captur…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）
Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully expl…
The abundant spatial and angular information from light fields has allowed the development of multiple disparity estimation approaches. However, the acquisition of light fields requires high storage and processing cost, limiting the use of this technology in practical applications. To overcome these drawbacks, the compressive sensing (CS) theory has allowed the development of optical architectures to acquire a single coded light field measurement. This measurement is decoded using an optimization algorithm or deep neural network that requires high computational costs. The traditional approach …
