While design of high performance lenses and image sensors has long been the focus of camera development, the size, weight and power of image data processing components is currently the primary barrier to radical improvements in camera resolution. Here we show that Deep-Learning- Aided Compressive Sampling (DLACS) can reduce operating power on camera-head electronics by 20x. Traditional compressive sampling has to date been primarily applied in the physical sensor layer, we show here that with aid from deep learning algorithms, compressive sampling offers unique power management advantages in digital layer compression.
We demonstrate an approach that allows taking videos at very high-speeds of over 100,000 frames per second (fps) by exploiting the fast sampling rate of the standard rolling-shutter readout mechanism, common to most conventional sensors, and a compressive-sampling acquisition scheme. Our approach is directly applied to a conventional imaging system by the simple addition of a diffuser to the pupil plane, randomly encoding the entire field-of-view to each camera row, while maintaining diffraction-limited resolution. A short video is reconstructed from a single camera frame via a compressed-sens…
We develop novel compressive coded rotating mirror (CCRM) camera to capture events at high frame rates in passive mode with a compact instrument design at the fraction of the cost compared to other high-speed imaging cameras. Operation of CCRM camera is based on the amplitude optical encoding (grey scale) and a continuous frame sweep across a low-cost detector using a motorized rotating mirror system which can achieve single pixel shift between adjacent frames. Amplitude encoding and continuous frame overlapping enable the CCRM camera to achieve high number of captured frames and high temporal…
The recently described pushframe imager, a parallelized single pixel camera capturing with a pushbroom-like motion, is intrinsically suited to both remote-sensing and compressive sampling. It optically applies a 2D mask to the imaged scene, before performing light integration along a single spatial axis, but previous work has not made use of the architecture's potential for taking measurements sparsely. In this paper we develop a strongly performing static binarized noiselet compressive sampling mask design, tailored to pushframe hardware, allowing both a single exposure per motion time-step, …
Adaptive block-based compressive sensing (ABCS) algorithms are studied in the context of the practical realization of compressive sensing on resource-constrained image and video sensing platforms that use single-pixel cameras, multi-pixel cameras or focal plane processing sensors. In this paper, we introduce two novel ABCS algorithms that are suitable for compressively sensing images or intra-coded video frames. Both use deterministic 2D-DCT dictionaries when sensing the images instead of random dictionaries. The first uses a low number of compressive measurements to compute the block boundary…
RIKEN（理化学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Tokyo（东京大学）；Nokia Bell Labs（诺基亚贝尔实验室）
Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) using a single coded two-dimensional (2D) measurement, in which algorithms are used to perform the inverse problem. Due to the ill-posed nature, various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. Unfortunately, the accuracy and computational complexity are unsatisfied. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurement, in thi…
Every day around the world, interminable terabytes of data are being captured for surveillance purposes. A typical 1-2MP CCTV camera generates around 7-12GB of data per day. Frame-by-frame processing of such enormous amount of data requires hefty computational resources. In recent years, compressive sensing approaches have shown impressive results in signal processing by reducing the sampling bandwidth. Different sampling mechanisms were developed to incorporate compressive sensing in image and video acquisition. Pixel-wise coded exposure is one among the promising sensing paradigms for captur…
Spectral camera based on ghost imaging via sparsity constraints (GISC spectral camera) obtains three-dimensional (3D) hyperspectral information with two-dimensional (2D) compressive measurements in a single shot, which has attracted much attention in recent years. However, its imaging quality and real-time performance of reconstruction still need to be further improved. Recently, deep learning has shown great potential in improving the reconstruction quality and reconstruction speed for computational imaging. When applying deep learning into GISC spectral camera, there are several challenges n…
We use convolutional neural networks to recover images optically down-sampled by $6.7\times$ using coherent aperture synthesis over a 16 camera array. Where conventional ptychography relies on scanning and oversampling, here we apply decompressive neural estimation to recover full resolution image from a single snapshot, although as shown in simulation multiple snapshots can be used to improve SNR. In place training on experimental measurements eliminates the need to directly calibrate the measurement system. We also present simulations of diverse array camera sampling strategies to explore ho…
Duke University（杜克大学）；University of Arizona（亚利桑那大学）
We demonstrate a physics-aware transformer for feature-based data fusion from cameras with diverse resolution, color spaces, focal planes, focal lengths, and exposure. We also demonstrate a scalable solution for synthetic training data generation for the transformer using open-source computer graphics software. We demonstrate image synthesis on arrays with diverse spectral responses, instantaneous field of view and frame rate.
Snapshot spectral imaging is rapidly gaining interest for remote sensing applications. Acquiring spatial and spectral data within one image promotes fast measurement times, and reduces the need for stabilized scanning imaging systems. Many current snapshot technologies, which rely on gratings or prisms to characterize wavelength information, are difficult to reduce in size for portable hyperspectral imaging. Here, we show that a multicore multimode fiber can be used as a compact spectral imager with sub-nanometer resolution, by encoding spectral information within a monochrome CMOS camera. We …
We describe the application of structured imaging with a single pixel camera to imaging through fog. We demonstrate the use of a high-pass filter on the detected bucket signals to suppress the effects of temporal variations of fog density and enable an effective reconstruction of the image. A quantitative analysis and comparison of several high-pass filters are demonstrated for the application. Both computational ghost imaging and compressive sensing techniques were used for image reconstruction and compressive sensing was observed to give a higher reconstructed image quality.
This article experimentally examines different configurations of a novel multi-camera array microscope (MCAM) imaging technology. The MCAM is based upon a densely packed array of "micro-cameras" to jointly image across a large field-of-view at high resolution. Each micro-camera within the array images a unique area of a sample of interest, and then all acquired data with 54 micro-cameras are digitally combined into composite frames, whose total pixel counts significantly exceed the pixel counts of standard microscope systems. We present results from three unique MCAM configurations for differe…
National University of Defense Technology（国防科技大学）；Sun Yat-sen University（中山大学）
Camera arrays provide spatial and angular information within a single snapshot. With refocusing methods, focal planes can be altered after exposure. In this letter, we propose a light field refocusing method to improve the imaging quality of camera arrays. In our method, the disparity is first estimated. Then, the unfocused region (bokeh) is rendered by using a depth-based anisotropic filter. Finally, the refocused image is produced by a reconstruction-based superresolution approach where the bokeh image is used as a regularization term. Our method can selectively refocus images with focused r…
Polarisation Filter Array (PFA) cameras allow the analysis of light polarisation state in a simple and cost-effective manner. Such filter arrays work as the Bayer pattern for colour cameras, sharing similar advantages and drawbacks. Among the others, the raw image must be demosaiced considering the local variations of the PFA and the characteristics of the imaged scene. Non-linear effects, like the cross-talk among neighbouring pixels, are difficult to explicitly model and suggest the potential advantage of a data-driven learning approach. However, the PFA cannot be removed from the sensor, ma…
CalibFPA: A Focal Plane Array Imaging System based on Online Deep-Learning Calibration
ASELSAN（ASELSAN公司）；Nanodev Scientific（Nanodev Scientific公司）；Ostim Technical University（奥斯蒂姆技术大学）；National Magnetic Resonance Research Center (UMRAM)（国家磁共振研究中心（UMRAM））；Bilkent University（比尔肯特大学）
This paper presents a method for capturing high-speed video using an asynchronous camera array. Our method sequentially fires each sensor in a camera array with a small time offset and assembles captured frames into a high-speed video according to the time stamps. The resulting video, however, suffers from parallax jittering caused by the viewpoint difference among sensors in the camera array. To address this problem, we develop a dedicated novel view synthesis algorithm that transforms the video frames as if they were captured by a single reference sensor. Specifically, for any frame from a n…
Security monitoring via ubiquitous cameras and their more extended in intelligent buildings stand to gain from advances in signal processing and machine learning. While these innovative and ground-breaking applications can be considered as a boon, at the same time they raise significant privacy concerns. In fact, recent GDPR (General Data Protection Regulation) legislation has highlighted and become an incentive for privacy-preserving solutions. Typical privacy-preserving video monitoring schemes address these concerns by either anonymizing the sensitive data. However, these approaches suffer …
