High dynamic range (HDR) images are important for a range of tasks, from navigation to consumer photography. Accordingly, a host of specialized HDR sensors have been developed, the most successful of which are based on capturing variable per-pixel exposures. In essence, these methods capture an entire exposure bracket sequence at once in a single shot. This paper presents a straightforward but highly effective approach for turning an off-the-shelf polarization camera into a high-performance HDR camera. By placing a linear polarizer in front of the polarization camera, we are able to simultaneously capture four images with varied exposures, which are determined by the orientation of the polarizer. We develop an outlier-robust and self-calibrating algorithm to reconstruct an HDR image (at a single polarity) from these measurements. Finally, we demonstrate the efficacy of our approach with extensive real-world experiments.
University of Alberta（阿尔伯塔大学）；NUDT（国防科技大学）；Huawei Fields Lab（华为Fields实验室）
The recent development of the on-chip micro-polarizer technology has made it possible to acquire four spatially aligned and temporally synchronized polarization images with the same ease of operation as a conventional camera. In this paper, we investigate the use of this sensor technology in high-dynamic-range (HDR) imaging. Specifically, observing that natural light can be attenuated differently by varying the orientation of the polarization filter, we treat the multiple images captured by the polarization camera as a set captured under different exposure times. In our approach, we first stud…
We use convolutional neural networks to recover images optically down-sampled by $6.7\times$ using coherent aperture synthesis over a 16 camera array. Where conventional ptychography relies on scanning and oversampling, here we apply decompressive neural estimation to recover full resolution image from a single snapshot, although as shown in simulation multiple snapshots can be used to improve SNR. In place training on experimental measurements eliminates the need to directly calibrate the measurement system. We also present simulations of diverse array camera sampling strategies to explore ho…
National Center for Data Analytics and Artificial Intelligence（国家数据分析与人工智能中心）；KACST（阿卜杜勒-阿齐兹国王科技城）；Visual Computing Center（视觉计算中心）；KAUST（阿卜杜拉国王科技大学）
This paper study the reconstruction of High Dynamic Range (HDR) video from snapshot-coded LDR video. Constructing an HDR video requires restoring the HDR values for each frame and maintaining the consistency between successive frames. HDR image acquisition from single image capture, also known as snapshot HDR imaging, can be achieved in several ways. For example, the reconfigurable snapshot HDR camera is realized by introducing an optical element into the optical stack of the camera; by placing a coded mask at a small standoff distance in front of the sensor. High-quality HDR image can be reco…
Nanjing University of Science and Technology（南京理工大学）
We consider using the system's optical imaging process with convolutional neural networks (CNNs) to solve the snapshot hyperspectral imaging reconstruction problem, which uses a dual-camera system to capture the three-dimensional hyperspectral images (HSIs) in a compressed way. Various methods using CNNs have been developed in recent years to reconstruct HSIs, but most of the supervised deep learning methods aimed to fit a brute-force mapping relationship between the captured compressed image and standard HSIs. Thus, the learned mapping would be invalid when the observation data deviate from t…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Humboldt-University（洪堡大学）
Spectral imaging technologies have rapidly evolved during the past decades. The recent development of single-camera-one-shot techniques for hyperspectral imaging allows multiple spectral bands to be captured simultaneously (3x3, 4x4 or 5x5 mosaic), opening up a wide range of applications. Examples include intraoperative imaging, agricultural field inspection and food quality assessment. To capture images across a wide spectrum range, i.e. to achieve high spectral resolution, the sensor design sacrifices spatial resolution. With increasing mosaic size, this effect becomes increasingly detriment…
A compressive sensing based circular polarization snapshot spectral imaging system is proposed in this paper to acquire two-dimensional spatial, one-dimensional circular polarization (the right and left circular polarization), and one-dimensional spectral information, simultaneously. Using snapshot can collect the entire four-dimensional datacube in a single integration period. The dispersion prism in the coded aperture snapshot spectral imager is replaced by the combination of an Amici prism and a Wollaston prism to implement the spectral shifting along two orthogonal directions, which greatl…
Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspec…
The convergence of recent advances in optical fabrication and digital processing yields a new generation of imaging technology: light-field cameras, which bridge the realms of applied mathematics, optics, and high-performance computing. Herein for the first time, we introduce the paradigm of light-field imaging into laryngoscopy. The resultant probe can image the three-dimensional (3D) shape of vocal folds within a single camera exposure. Furthermore, to improve the spatial resolution, we developed an image fusion algorithm, providing a simple solution to a long-standing problem in light-field…
By encoding the high-dimensional light-field imaging information into a detectable two-dimensional speckle plane, ghost imaging camera via sparsity constraints (GISC camera) can directly catch the high-dimensional light-field imaging information with only one snapshot. This makes it worth to revisit the spatial resolution limit of this optical imaging system. In this paper we show both theoretically and experimentally that, while the resolution in high-dimensional light-field space is still limited by diffraction, the statistical spatial resolution of GISC camera can be greatly improved compar…
This paper describes a multimodal vision sensor that integrates three types of cameras, including a stereo camera, a polarization camera and a panoramic camera. Each sensor provides a specific dimension of information: the stereo camera measures depth per pixel, the polarization obtains the degree of polarization, and the panoramic camera captures a 360-degree landscape. Data fusion and advanced environment perception could be built upon the combination of sensors. Designed especially for autonomous driving, this vision sensor is shipped with a robust semantic segmentation network. In addition…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Snapshot compressive imaging (SCI) aims to record three-dimensional signals via a two-dimensional camera. For the sake of building a fast and accurate SCI recovery algorithm, we incorporate the interpretability of model-based methods and the speed of learning-based ones and present a novel dense deep unfolding network (DUN) with 3D-CNN prior for SCI, where each phase is unrolled from an iteration of Half-Quadratic Splitting (HQS). To better exploit the spatial-temporal correlation among frames and address the problem of information loss between adjacent phases in existing DUNs, we propose to a…
We propose a novel concept of differential coded aperture snapshot spectral imaging (D-CASSI) technique exploiting the benefits of using {-1,+1} random mask, which is demonstrated by a broadband single-snapshot hyperspectral camera using compressed sensing. To double the information, we encode the image by two complementary random masks, which proved to be superior to two independent patterns. We utilize dispersed and non-dispersed encoded images captured in parallel onto a single detector. We explored several different approaches to processing the measured data, which demonstrates significant…
Visual discrimination of clinical tissue types remains challenging, with traditional RGB imaging providing limited contrast for such tasks. Hyperspectral imaging (HSI) is a promising technology providing rich spectral information that can extend far beyond three-channel RGB imaging. Moreover, recently developed snapshot HSI cameras enable real-time imaging with significant potential for clinical applications. Despite this, the investigation into the relative performance of HSI over RGB imaging for semantic segmentation purposes has been limited, particularly in the context of medical imaging. …
Bell Labs（贝尔实验室）；College of Optical Sciences, University of Arizona（亚利桑那大学光学科学学院）；Northwestern University（西北大学）
Capturing high-dimensional (HD) data is a long-term challenge in signal processing and related fields. Snapshot compressive imaging (SCI) uses a two-dimensional (2D) detector to capture HD ($\ge3$D) data in a {\em snapshot} measurement. Via novel optical designs, the 2D detector samples the HD data in a {\em compressive} manner; following this, algorithms are employed to reconstruct the desired HD data-cube. SCI has been used in hyperspectral imaging, video, holography, tomography, focal depth imaging, polarization imaging, microscopy, \etc.~Though the hardware has been investigated for more t…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Hyperspectral imaging is an essential imaging modality for a wide range of applications, especially in remote sensing, agriculture, and medicine. Inspired by existing hyperspectral cameras that are either slow, expensive, or bulky, reconstructing hyperspectral images (HSIs) from a low-budget snapshot measurement has drawn wide attention. By mapping a truncated numerical optimization algorithm into a network with a fixed number of phases, recent deep unfolding networks (DUNs) for spectral snapshot compressive sensing (SCI) have achieved remarkable success. However, DUNs are far from reaching th…
University of California, Berkeley（加利福尼亚大学伯克利分校）；UCB/UCSF Joint Graduate Program in Bioengineering（加州大学伯克利分校/加州大学旧金山分校联合生物工程研究生项目）
Hyperspectral imaging is useful for applications ranging from medical diagnostics to agricultural crop monitoring; however, traditional scanning hyperspectral imagers are prohibitively slow and expensive for widespread adoption. Snapshot techniques exist but are often confined to bulky benchtop setups or have low spatio-spectral resolution. In this paper, we propose a novel, compact, and inexpensive computational camera for snapshot hyperspectral imaging. Our system consists of a tiled spectral filter array placed directly on the image sensor and a diffuser placed close to the sensor. Each poi…
We demonstrate single-shot compressive three-dimensional (3D) $(x, y, z)$ imaging based on interference coding. The depth dimension of the object is encoded into the interferometric spectra of the light field, resulting a $(x, y, λ)$ datacube which is subsequently measured by a single-shot spectrometer. By implementing a compression ratio up to $400$, we are able to reconstruct $1G$ voxels from a 2D measurement. Both an optimization based compressive sensing algorithm and a deep learning network are developed for 3D reconstruction from a single 2D coded measurement. Due to the fast acquisition…
National University of Defense Technology（国防科技大学）；Sun Yat-sen University（中山大学）
Camera arrays provide spatial and angular information within a single snapshot. With refocusing methods, focal planes can be altered after exposure. In this letter, we propose a light field refocusing method to improve the imaging quality of camera arrays. In our method, the disparity is first estimated. Then, the unfocused region (bokeh) is rendered by using a depth-based anisotropic filter. Finally, the refocused image is produced by a reconstruction-based superresolution approach where the bokeh image is used as a regularization term. Our method can selectively refocus images with focused r…
This paper proposes a novel learning based high-dynamic-range (HDR) reconstruction method using a polarization camera. We utilize a previous observation that polarization filters with different orientations can attenuate natural light differently, and we treat the multiple images acquired by the polarization camera as a set acquired under different exposure times, to introduce the development of solutions for the HDR reconstruction problem. We propose a deep HDR reconstruction framework with a feature masking mechanism that uses polarimetric cues available from the polarization camera, called …
Snapshot spectral imaging is rapidly gaining interest for remote sensing applications. Acquiring spatial and spectral data within one image promotes fast measurement times, and reduces the need for stabilized scanning imaging systems. Many current snapshot technologies, which rely on gratings or prisms to characterize wavelength information, are difficult to reduce in size for portable hyperspectral imaging. Here, we show that a multicore multimode fiber can be used as a compact spectral imager with sub-nanometer resolution, by encoding spectral information within a monochrome CMOS camera. We …
Tsinghua University（清华大学）；Massachusetts Institute of Technology（麻省理工学院）；Bell Labs（贝尔实验室）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
High resolution images are widely used in our daily life, whereas high-speed video capture is challenging due to the low frame rate of cameras working at the high resolution mode. Digging deeper, the main bottleneck lies in the low throughput of existing imaging systems. Towards this end, snapshot compressive imaging (SCI) was proposed as a promising solution to improve the throughput of imaging systems by compressive sampling and computational reconstruction. During acquisition, multiple high-speed images are encoded and collapsed to a single measurement. After this, algorithms are employed t…
Depth imaging is vital for many emerging technologies with applications in augmented reality, robotics, gesture detection, and facial recognition. These applications, however, demand compact and low-power systems beyond the capabilities of state-of-the-art depth cameras. Here, we leverage ultrathin dielectric metasurfaces to demonstrate a solution that, with a single surface, replicates the functionality of a high-performance depth camera typically comprising a spatial light modulator, polarizer, and three lenses. Using cylindrical nano-scatterers that can arbitrarily modify the phase of an in…
PIDSR: Complementary Polarized Image Demosaicing and Super-Resolution
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；National Institute of Informatics（国家信息研究所）；State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理国家重点实验室）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学计算机科学学院视觉技术国家工程研究中心）
RIKEN（理化学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Tokyo（东京大学）；Nokia Bell Labs（诺基亚贝尔实验室）
Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) using a single coded two-dimensional (2D) measurement, in which algorithms are used to perform the inverse problem. Due to the ill-posed nature, various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. Unfortunately, the accuracy and computational complexity are unsatisfied. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurement, in thi…
Video Snapshot compressive imaging (SCI) is a promising technique to capture high-speed videos, which transforms the imaging speed from the detector to mask modulating and only needs a single measurement to capture multiple frames. The algorithm to reconstruct high-speed frames from the measurement plays a vital role in SCI. In this paper, we consider the promising reconstruction algorithm framework, namely plug-and-play (PnP), which is flexible to the encoding process comparing with other deep learning networks. One drawback of existing PnP algorithms is that they use a pre-trained denoising …
This paper presents a method of passive non-line-of-sight (NLOS) imaging using polarization cues. A key observation is that the oblique light has a different polarimetric signal. It turns out this effect is due to the polarization axis rotation, a phenomena which can be used to better condition the light transport matrix for non-line-of-sight imaging. Our analysis and results show that the use of a polarizer in front of the camera is not only a separate technique, but it can be seen as an enhancement technique for more advanced forms of passive NLOS imaging. For example, this paper shows that …
Fundus photography is indispensable for clinical detection and management of eye diseases. Limited image contrast and field of view (FOV) are common limitations of conventional fundus cameras, making it difficult to detect subtle abnormalities at the early stages of eye diseases. Further improvements of image contrast and FOV coverage are important to improve early disease detection and reliable treatment assessment. We report here a portable fundus camera, with a wide FOV and high dynamic range (HDR) imaging capabilities. Miniaturized indirect ophthalmoscopy illumination was employed to achie…
This article experimentally examines different configurations of a novel multi-camera array microscope (MCAM) imaging technology. The MCAM is based upon a densely packed array of "micro-cameras" to jointly image across a large field-of-view at high resolution. Each micro-camera within the array images a unique area of a sample of interest, and then all acquired data with 54 micro-cameras are digitally combined into composite frames, whose total pixel counts significantly exceed the pixel counts of standard microscope systems. We present results from three unique MCAM configurations for differe…
Coded aperture snapshot spectral imaging (CASSI) makes it possible to recover 3D hyperspectral data from a single 2D image. However, the reconstruction problem is severely underdetermined and efforts to improve the compression ratio typically make the imaging system more complex and cause a significant loss of incoming light intensity. In this paper, we propose a novel approach to CASSI which enables capturing both spectrally sheared and integrated image of a scene with a single camera. We performed hyperspectral imaging of three different testing scenes in the spectral range of 500-900 nm. We…
