Coded aperture compressive temporal imaging

Data Driven Coded Aperture Design for Depth Recovery

Inserting a patterned occluder at the aperture of a camera lens has been shown to improve the recovery of depth map and all-focus image compared to a fully open aperture. However, design of the aperture pattern plays a very critical role. Previous approaches for designing aperture codes make simple assumptions on image distributions to obtain metrics for evaluating aperture codes. However, real images may not follow those assumptions and hence the designed code may not be optimal for them. To address this drawback we propose a data driven approach for learning the optimal aperture pattern to r…

Privacy-Preserving Action Recognition using Coded Aperture Videos

The risk of unauthorized remote access of streaming video from networked cameras underlines the need for stronger privacy safeguards. We propose a lens-free coded aperture camera system for human action recognition that is privacy-preserving. While coded aperture systems exist, we believe ours is the first system designed for action recognition without the need for image restoration as an intermediate step. Action recognition is done using a deep network that takes in as input, non-invertible motion features between pairs of frames computed using phase correlation and log-polar transformation.…

编码的E2LF：从事件中获取4D光场

Coded-E2LF: Coded Aperture Light Field Imaging from Events

本文提出Coded-E2LF方法，通过编码孔径和事件-only相机实现4D光场的像素级精确重建，首次证明仅使用事件数据即可获得高精度光场。

A Unified Framework for Compressive Video Recovery from Coded Exposure Techniques

Several coded exposure techniques have been proposed for acquiring high frame rate videos at low bandwidth. Most recently, a Coded-2-Bucket camera has been proposed that can acquire two compressed measurements in a single exposure, unlike previously proposed coded exposure techniques, which can acquire only a single measurement. Although two measurements are better than one for an effective video recovery, we are yet unaware of the clear advantage of two measurements, either quantitatively or qualitatively. Here, we propose a unified learning-based framework to make such a qualitative and quan…

FlatCam: Thin, Bare-Sensor Cameras using Coded Aperture and Computation

FlatCam is a thin form-factor lensless camera that consists of a coded mask placed on top of a bare, conventional sensor array. Unlike a traditional, lens-based camera where an image of the scene is directly recorded on the sensor pixels, each pixel in FlatCam records a linear combination of light from multiple scene elements. A computational algorithm is then used to demultiplex the recorded measurements and reconstruct an image of the scene. FlatCam is an instance of a coded aperture imaging system; however, unlike the vast majority of related work, we place the coded mask extremely close to…

编码孔径与扩散后验采样的单图像散焦深度估计

Single-Image Depth from Defocus with Coded Aperture and Diffusion Posterior Sampling

提出一种基于编码孔径和扩散后验采样的单快照散焦深度重建方法，利用扩散先验作为正则化，无需配对训练数据，在仿真和原型相机上优于U-Net基线和经典DFD方法。

10-mega pixel snapshot compressive imaging with a hybrid coded aperture

High resolution images are widely used in our daily life, whereas high-speed video capture is challenging due to the low frame rate of cameras working at the high resolution mode. Digging deeper, the main bottleneck lies in the low throughput of existing imaging systems. Towards this end, snapshot compressive imaging (SCI) was proposed as a promising solution to improve the throughput of imaging systems by compressive sampling and computational reconstruction. During acquisition, multiple high-speed images are encoded and collapsed to a single measurement. After this, algorithms are employed t…

Lensless Imaging by Compressive Sensing

In this paper, we propose a lensless compressive imaging architecture. The architecture consists of two components, an aperture assembly and a sensor. No lens is used. The aperture assembly consists of a two dimensional array of aperture elements. The transmittance of each aperture element is independently controllable. The sensor is a single detection element. A compressive sensing matrix is implemented by adjusting the transmittance of the individual aperture elements according to the values of the sensing matrix. The proposed architecture is simple and reliable because no lens is used. The …

EDoF-NeRF: 使用编码孔径相机扩展景深的神经辐射场

EDoF-NeRF: extended depth-of-field neural radiance fields using a coded aperture camera

提出一种通过编码孔径相机扩展景深的方法，构建高保真神经辐射场，实现从不同视角图像渲染新视图，并验证其优于传统孔径相机。

Progressive Content-aware Coded Hyperspectral Compressive Imaging

Hyperspectral imaging plays a pivotal role in a wide range of applications, like remote sensing, medicine, and cytology. By acquiring 3D hyperspectral images (HSIs) via 2D sensors, the coded aperture snapshot spectral imaging (CASSI) has achieved great success due to its hardware-friendly implementation and fast imaging speed. However, for some less spectrally sparse scenes, single snapshot and unreasonable coded aperture design tend to make HSI recovery more ill-posed and yield poor spatial and spectral fidelity. In this paper, we propose a novel Progressive Content-Aware CASSI framework, dub…

Compressive Holographic Video

Compressed sensing has been discussed separately in spatial and temporal domains. Compressive holography has been introduced as a method that allows 3D tomographic reconstruction at different depths from a single 2D image. Coded exposure is a temporal compressed sensing method for high speed video acquisition. In this work, we combine compressive holography and coded exposure techniques and extend the discussion to 4D reconstruction in space and time from one coded captured image. In our prototype, digital in-line holography was used for imaging macroscopic, fast moving objects. The pixel-wise…

使用学习型彩色编码光阑从单张光学编码图像中进行深度估计

Depth Estimation from a Single Optical Encoded Image using a Learned Colored-Coded Aperture

提出一种具有丰富光谱信息的彩色编码光阑（CCA）并联合CNN进行端到端优化，从单张光学编码图像中检索深度信息，在三个数据集上优于现有方法并通过低成本原型验证了真实场景效果。

A Compressive Sensing Video dataset using Pixel-wise coded exposure

Manifold amount of video data gets generated every minute as we read this document, ranging from surveillance to broadcasting purposes. There are two roadblocks that restrain us from using this data as such, first being the storage which restricts us from only storing the information based on the hardware constraints. Secondly, the computation required to process this data is highly expensive which makes it infeasible to work on them. Compressive sensing(CS)[2] is a signal process technique[11], through optimization, the sparsity of a signal can be exploited to recover it from far fewer sample…

Signal to Noise Ratio in Lensless Compressive Imaging

We analyze the signal to noise ratio (SNR) in a lensless compressive imaging (LCI) architecture. The architecture consists of a sensor of a single detecting element and an aperture assembly of an array of programmable elements. LCI can be used in conjunction with compressive sensing to capture images in a compressed form of compressive measurements. In this paper, we perform SNR analysis of the LCI and compare it with imaging with a pinhole or a lens. We will show that the SNR in the LCI is independent of the image resolution, while the SNR in either pinhole aperture imaging or lens aperture i…

Learning to Estimate Kernel Scale and Orientation of Defocus Blur with Asymmetric Coded Aperture

Consistent in-focus input imagery is an essential precondition for machine vision systems to perceive the dynamic environment. A defocus blur severely degrades the performance of vision systems. To tackle this problem, we propose a deep-learning-based framework estimating the kernel scale and orientation of the defocus blur to adjust lens focus rapidly. Our pipeline utilizes 3D ConvNet for a variable number of input hypotheses to select the optimal slice from the input stack. We use random shuffle and Gumbel-softmax to improve network performance. We also propose to generate synthetic defocuse…

On Space-spectrum Uncertainty Analysis for Coded Aperture Systems

We introduce and analyze the concept of space-spectrum uncertainty for certain commonly-used designs for spectrally programmable cameras. Our key finding states that, it is impossible to simultaneously capture high-resolution spatial images while programming the spectrum at high resolution. This phenomenon arises due to a Fourier relationship between the aperture used for obtaining spectrum and its corresponding diffraction blur in the (spatial) image. We show that the product of spatial and spectral standard deviations is lower bounded by λ/4π{ν_0} femto square-meters, where {ν_0} is the dens…

Image and Depth from a Single Defocused Image Using Coded Aperture Photography

Depth from defocus and defocus deblurring from a single image are two challenging problems that are derived from the finite depth of field in conventional cameras. Coded aperture imaging is one of the techniques that is used for improving the results of these two problems. Up to now, different methods have been proposed for improving the results of either defocus deblurring or depth estimation. In this paper, a multi-objective function is proposed for evaluating and designing aperture patterns with the aim of improving the results of both depth from defocus and defocus deblurring. Pattern eval…

Compressive Light Field Reconstructions using Deep Learning

Light field imaging is limited in its computational processing demands of high sampling for both spatial and angular dimensions. Single-shot light field cameras sacrifice spatial resolution to sample angular viewpoints, typically by multiplexing incoming rays onto a 2D sensor array. While this resolution can be recovered using compressive sensing, these iterative solutions are slow in processing a light field. We present a deep learning approach using a new, two branch network architecture, consisting jointly of an autoencoder and a 4D CNN, to recover a high resolution 4D light field from a si…

Scene-adaptive Coded Apertures Imaging

Coded aperture imaging systems have recently shown great success in recovering scene depth and extending the depth-of-field. The ideal pattern, however, would have to serve two conflicting purposes: 1) be broadband to ensure robust deconvolution and 2) has sufficient zero-crossings for a high depth discrepancy. This paper presents a simple but effective scene-adaptive coded aperture solution to bridge this gap. We observe that the geometric structures in a natural scene often exhibit only a few edge directions, and the successive frames are closely correlated. Therefore we adopt a spatial part…

A new compressive video sensing framework for mobile broadcast

A new video coding method based on compressive sampling is proposed. In this method, a video is coded using compressive measurements on video cubes. Video reconstruction is performed by minimization of total variation (TV) of the pixelwise DCT coefficients along the temporal direction. A new reconstruction algorithm is developed from TVAL3, an efficient TV minimization algorithm based on the alternating minimization and augmented Lagrangian methods. Video coding with this method is inherently scalable, and has applications in mobile broadcast.

使用编码孔径与事件的高时间效率光场采集

Time-Efficient Light-Field Acquisition Using Coded Aperture and Events

本文提出将编码孔径序列与事件相机结合，在单次曝光内联合图像帧和事件重建光场，并设计了可端到端训练的深度光学算法流程，实验证明其重建精度优于多种单曝光成像方法。

被动快照编码孔径双像素RGB-D成像

Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

本文提出CADS编码孔径双像素感知方法，通过端到端联合学习最优编码模式和重建算法，在被动快照紧凑条件下显著提升RGB-D深度估计与全聚焦重建精度。

基于稀疏性与深度图像先验的编码孔径快照光谱成像快速交替最小化算法

A Fast Alternating Minimization Algorithm for Coded Aperture Snapshot Spectral Imaging Based on Sparsity and Deep Image Priors

针对CASSI高光谱重建在少测量、多通道下的不适定问题，提出融合稀疏性与DIP的无训练快速交替最小化算法Fama-SDIP，并在仿真和真实数据集上取得领先性能。

基于知识蒸馏方法的高度受限编码孔径成像系统设计

Highly Constrained Coded Aperture Imaging Systems Design Via a Knowledge Distillation Approach

本文提出知识蒸馏框架，用无约束教师系统指导受物理约束的学生系统优化，在编码孔径单像素相机上验证，优于端到端方法。

Optimizing Codes for Source Separation in Color Image Demosaicing and Compressive Video Recovery

There exist several applications in image processing (eg: video compressed sensing [Hitomi, Y. et al, "Video from a single coded exposure photograph using a learned overcomplete dictionary"] and color image demosaicing [Moghadam, A. A. et al, "Compressive Framework for Demosaicing of Natural Images"]) which require separation of constituent images given measurements in the form of a coded superposition of those images. Physically practical code patterns in these applications are non-negative, systematically structured, and do not always obey the nice incoherence properties of other patterns su…

FPA-CS: Focal Plane Array-based Compressive Imaging in Short-wave Infrared

Cameras for imaging in short and mid-wave infrared spectra are significantly more expensive than their counterparts in visible imaging. As a result, high-resolution imaging in those spectrum remains beyond the reach of most consumers. Over the last decade, compressive sensing (CS) has emerged as a potential means to realize inexpensive short-wave infrared cameras. One approach for doing this is the single-pixel camera (SPC) where a single detector acquires coded measurements of a high-resolution image. A computational reconstruction algorithm is then used to recover the image from these coded …

Joint System and Algorithm Design for Computationally Efficient Fan Beam Coded Aperture X-ray Coherent Scatter Imaging

In x-ray coherent scatter tomography, tomographic measurements of the forward scatter distribution are used to infer scatter densities within a volume. A radiopaque 2D pattern placed between the object and the detector array enables the disambiguation between different scatter events. The use of a fan beam source illumination to speed up data acquisition relative to a pencil beam presents computational challenges. To facilitate the use of iterative algorithms based on a penalized Poisson log-likelihood function, efficient computational implementation of the forward and backward models are need…

Compressive Hyperspectral Imaging with Side Information

A blind compressive sensing algorithm is proposed to reconstruct hyperspectral images from spectrally-compressed measurements.The wavelength-dependent data are coded and then superposed, mapping the three-dimensional hyperspectral datacube to a two-dimensional image. The inversion algorithm learns a dictionary {\em in situ} from the measurements via global-local shrinkage priors. By using RGB images as side information of the compressive sensing system, the proposed approach is extended to learn a coupled dictionary from the joint dataset of the compressed measurements and the corresponding RG…

Spatial Sparse subspace clustering for Compressive Spectral imaging

This paper aims at developing a clustering approach with spectral images directly from CASSI compressive measurements. The proposed clustering method first assumes that compressed measurements lie in the union of multiple low-dimensional subspaces. Therefore, sparse subspace clustering (SSC) is an unsupervised method that assigns compressed measurements to their respective subspaces. In addition, a 3D spatial regularizer is added into the SSC problem, thus taking full advantages of the spatial information contained in spectral images. The performance of the proposed spectral image clustering a…

基于概率的四维光场特征嵌入用于压缩成像与去噪

Probabilistic-based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising

本文提出基于概率的特征嵌入（PFE）方法，通过在概率空间组装低维卷积模式捕获四维光场空间-角度信息，并分别构建循环一致重建网络和迭代优化框架，在压缩成像与去噪任务上显著优于现有方法。

Fast Hyperspectral Image Recovery via Non-iterative Fusion of Dual-Camera Compressive Hyperspectral Imaging

Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) using a single coded two-dimensional (2D) measurement, in which algorithms are used to perform the inverse problem. Due to the ill-posed nature, various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. Unfortunately, the accuracy and computational complexity are unsatisfied. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurement, in thi…

CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo

Conventional stereo suffers from a fundamental trade-off between imaging volume and signal-to-noise ratio (SNR) -- due to the conflicting impact of aperture size on both these variables. Inspired by the extended depth of field cameras, we propose a novel end-to-end learning-based technique to overcome this limitation, by introducing a phase mask at the aperture plane of the cameras in a stereo imaging system. The phase mask creates a depth-dependent point spread function, allowing us to recover sharp image texture and stereo correspondence over a significantly extended depth of field (EDOF) th…

Lensless Compressive Imaging

We develop a lensless compressive imaging architecture, which consists of an aperture assembly and a single sensor, without using any lens. An anytime algorithm is proposed to reconstruct images from the compressive measurements; the algorithm produces a sequence of solutions that monotonically converge to the true signal (thus, anytime). The algorithm is developed based on the sparsity of local overlapping patches (in the transformation domain) and state-of-the-art results have been obtained. Experiments on real data demonstrate that encouraging results are obtained by measuring about 10% (of…

Low-Cost Compressive Sensing for Color Video and Depth

A simple and inexpensive (low-power and low-bandwidth) modification is made to a conventional off-the-shelf color video camera, from which we recover {multiple} color frames for each of the original measured frames, and each of the recovered frames can be focused at a different depth. The recovery of multiple frames for each measured frame is made possible via high-speed coding, manifested via translation of a single coded aperture; the inexpensive translation is constituted by mounting the binary code on a piezoelectric device. To simultaneously recover depth information, a {liquid} lens is m…

Unsupervised Spatial-spectral Network Learning for Hyperspectral Compressive Snapshot Reconstruction

Hyperspectral compressive imaging takes advantage of compressive sensing theory to achieve coded aperture snapshot measurement without temporal scanning, and the entire three-dimensional spatial-spectral data is captured by a two-dimensional projection during a single integration period. Its core issue is how to reconstruct the underlying hyperspectral image using compressive sensing reconstruction algorithms. Due to the diversity in the spectral response characteristics and wavelength range of different spectral imaging devices, previous works are often inadequate to capture complex spectral …

FPGA system for real-time computational extended depth of field imaging using phase aperture coding

We present a proof-of-concept end-to-end system for computational extended depth of field (EDOF) imaging. The acquisition is performed through a phase-coded aperture implemented by placing a thin wavelength-dependent optical mask inside the pupil of a conventional camera lens, as a result of which, each color channel is focused at a different depth. The reconstruction process receives the raw Bayer image as the input, and performs blind estimation of the output color image in focus at an extended range of depths using a patch-wise sparse prior. We present a fast non-iterative reconstruction al…

Deep Gaussian Scale Mixture Prior for Spectral Compressive Imaging

In coded aperture snapshot spectral imaging (CASSI) system, the real-world hyperspectral image (HSI) can be reconstructed from the captured compressive image in a snapshot. Model-based HSI reconstruction methods employed hand-crafted priors to solve the reconstruction problem, but most of which achieved limited success due to the poor representation capability of these hand-crafted priors. Deep learning based methods learning the mappings between the compressive images and the HSIs directly achieved much better results. Yet, it is nontrivial to design a powerful deep network heuristically for …

面向光谱压缩成像的渐进式流启发展开

Progressive Flow-inspired Unfolding for Spectral Compressive Imaging

针对CASSI重建中深度展开网络轨迹不可控导致质量突变的问题，提出受扩散和流匹配启发的轨迹可控展开框架，结合高效空间-光谱Transformer与频域融合，实现平滑连续优化，在质量与效率上超越现有方法。

Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging

In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradati…

面向快照压缩成像的硬件-提示协同学习

Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging

针对快照压缩成像模型跨硬件适配差、集中训练存隐私与异构性问题，提出FedHP框架，通过硬件条件提示器对齐客户端数据分布，在异构设置下性能优于主流FL框架0.35dB，并构建了快照光谱异构数据集。

通过色度-强度分解实现光谱压缩成像

Spectral Compressive Imaging via Chromaticity-Intensity Decomposition

本文提出了一种色度-强度分解框架，通过双相机CASSI系统实现超光谱图像的高效重建，结合混合Transformer和噪声估计模块，提升光谱和色度保真度。

LRDUN: 一种低秩深度展开网络用于高效光谱压缩成像

LRDUN: A Low-Rank Deep Unfolding Network for Efficient Spectral Compressive Imaging

LRDUN通过整合低秩分解与传感模型，提出一种低秩深度展开网络，以高效重建光谱压缩成像，减少计算成本并提升重建质量。

10K FPS下的事件增强快照压缩摄像技术

Event-Enhanced Snapshot Compressive Videography at 10K FPS

针对传统视频快照压缩成像丢失帧间动态信息的问题，提出融合事件相机的强度+事件混合成像方案与双分支Transformer重建方法，实现了24 FPS低成本传感器下0.1ms间隔的高质量高速摄像。

用于大视场RGBD成像的可学习离孔径编码

Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging

针对大视场下端到端成像保真度低的问题，提出离孔径放置DOE结合折衍混合建模的方法，提升了成像质量并验证了有效性。

残差退化学习展开框架与跨光谱和空间混合先验用于压缩光谱成像

Residual Degradation Learning Unfolding Framework with Mixing Priors across Spectral and Spatial for Compressive Spectral Imaging

本文提出残差退化学习展开框架（RDLUF）与跨光谱空间混合先验的Mix$S^2$ Transformer，用于压缩光谱成像，解决传感矩阵与真实退化不匹配问题，提升重建性能。

自监督单步扩散细化用于快照压缩成像

Self-Supervised One-Step Diffusion Refinement for Snapshot Compressive Imaging

提出自监督单步扩散框架，用于提升快照压缩成像的重建精度与效率，实现高质量图像生成并减少计算时间。

细节至关重要：受 Mamba 启发的快照光谱压缩成像联合展开网络

Detail Matters: Mamba-Inspired Joint Unfolding Network for Snapshot Spectral Compressive Imaging

本文提出 MiJUN，将物理展开与 Mamba 注意力及张量 mode-k 展开结合，用于快照光谱压缩成像中的高光谱重建，显著提升细节表示与重建性能。