This paper introduces a single-pixel HyperSpectral (HS) imaging framework based on Fourier Transform Interferometry (FTI). By combining a space-time coding of the light illumination with partial interferometric observations of a collimated light beam (observed by a single pixel), our system benefits from (i) reduced measurement rate and light-exposure of the observed object compared to common (Nyquist) FTI imagers, and (ii) high spectral resolution as desirable in, e.g., Fluorescence Spectroscopy (FS). From the principles of compressive sensing with multilevel sampling, our method leverages the sparsity "in level" of FS data, both in the spectral and the spatial domains. This allows us to optimize the space-time light coding using time-modulated Hadamard patterns. We confirm the effectiveness of our approach by a few numerical experiments.
A blind compressive sensing algorithm is proposed to reconstruct hyperspectral images from spectrally-compressed measurements.The wavelength-dependent data are coded and then superposed, mapping the three-dimensional hyperspectral datacube to a two-dimensional image. The inversion algorithm learns a dictionary {\em in situ} from the measurements via global-local shrinkage priors. By using RGB images as side information of the compressive sensing system, the proposed approach is extended to learn a coupled dictionary from the joint dataset of the compressed measurements and the corresponding RG…
Case Western Reserve University（凯斯西储大学）；Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO））；University of Queensland（昆士兰大学）；University of New South Wales（新南威尔士大学）
In this paper, we propose a new sampling strategy for hyperspectral signals that is based on dictionary learning and singular value decomposition (SVD). Specifically, we first learn a sparsifying dictionary from training spectral data using dictionary learning. We then perform an SVD on the dictionary and use the first few left singular vectors as the rows of the measurement matrix to obtain the compressive measurements for reconstruction. The proposed method provides significant improvement over the conventional compressive sensing approaches. The reconstruction performance is further improve…
Nanjing University of Information Science and Technology（南京信息工程大学）；National University of Singapore（新加坡国立大学）
Hyperspectral compressive imaging takes advantage of compressive sensing theory to achieve coded aperture snapshot measurement without temporal scanning, and the entire three-dimensional spatial-spectral data is captured by a two-dimensional projection during a single integration period. Its core issue is how to reconstruct the underlying hyperspectral image using compressive sensing reconstruction algorithms. Due to the diversity in the spectral response characteristics and wavelength range of different spectral imaging devices, previous works are often inadequate to capture complex spectral …
Tsinghua University（清华大学）；Nokia Bell Labs（诺基亚贝尔实验室）；Duke University（杜克大学）
Snapshot compressive imaging (SCI) refers to compressive imaging systems where multiple frames are mapped into a single measurement, with video compressive imaging and hyperspectral compressive imaging as two representative applications. Though exciting results of high-speed videos and hyperspectral images have been demonstrated, the poor reconstruction quality precludes SCI from wide applications.This paper aims to boost the reconstruction quality of SCI via exploiting the high-dimensional structure in the desired signal. We build a joint model to integrate the nonlocal self-similarity of vid…
Compressed Sensing (CS) is suitable for remote acquisition of hyperspectral images for earth observation, since it could exploit the strong spatial and spectral correlations, llowing to simplify the architecture of the onboard sensors. Solutions proposed so far tend to decouple spatial and spectral dimensions to reduce the complexity of the reconstruction, not taking into account that onboard sensors progressively acquire spectral rows rather than acquiring spectral channels. For this reason, we propose a novel progressive CS architecture based on separate sensing of spectral rows and joint re…
RIKEN（理化学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Tokyo（东京大学）；Nokia Bell Labs（诺基亚贝尔实验室）
Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) using a single coded two-dimensional (2D) measurement, in which algorithms are used to perform the inverse problem. Due to the ill-posed nature, various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. Unfortunately, the accuracy and computational complexity are unsatisfied. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurement, in thi…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Hyperspectral imaging is an essential imaging modality for a wide range of applications, especially in remote sensing, agriculture, and medicine. Inspired by existing hyperspectral cameras that are either slow, expensive, or bulky, reconstructing hyperspectral images (HSIs) from a low-budget snapshot measurement has drawn wide attention. By mapping a truncated numerical optimization algorithm into a network with a fixed number of phases, recent deep unfolding networks (DUNs) for spectral snapshot compressive sensing (SCI) have achieved remarkable success. However, DUNs are far from reaching th…
Colorado State University（科罗拉多州立大学）；Physical Sciences Inc.（物理科学公司）
Compressive sensing (CS) is a method of sampling which permits some classes of signals to be reconstructed with high accuracy even when they were under-sampled. In this paper we explore a phenomenon in which bandwise CS sampling of a hyperspectral data cube followed by reconstruction can actually result in amplification of chemical signals contained in the cube. Perhaps most surprisingly, chemical signal amplification generally seems to increase as the level of sampling decreases. In some examples, the chemical signal is significantly stronger in a data cube reconstructed from 10% CS sampling …
University of Oxford（牛津大学）；Ludwig–Maximilians–Universität München（慕尼黑大学）；John Adams Institute for Accelerator Science（约翰·亚当斯加速器科学研究所）
Presented is a novel way to combine snapshot compressive imaging and lateral shearing interferometry in order to capture the spatio-spectral phase of an ultrashort laser pulse in a single shot. A deep unrolling algorithm is utilised for the snapshot compressive imaging reconstruction due to its parameter efficiency and superior speed relative to other methods, potentially allowing for online reconstruction. The algorithm's regularisation term is represented using neural network with 3D convolutional layers, to exploit the spatio-spectral correlations that exist in laser wavefronts. Compressed …
Dual Hyperspectral Mamba for Efficient Spectral Compressive Imaging
The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Hunan University（湖南大学）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；Peking University（北京大学）
Hyperspectral imaging plays a pivotal role in a wide range of applications, like remote sensing, medicine, and cytology. By acquiring 3D hyperspectral images (HSIs) via 2D sensors, the coded aperture snapshot spectral imaging (CASSI) has achieved great success due to its hardware-friendly implementation and fast imaging speed. However, for some less spectrally sparse scenes, single snapshot and unreasonable coded aperture design tend to make HSI recovery more ill-posed and yield poor spatial and spectral fidelity. In this paper, we propose a novel Progressive Content-Aware CASSI framework, dub…
Key Laboratory of Photoelectronic Imaging Technology and System of Ministry of Education of China, School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院光电成像技术与系统教育部重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（特拉华大学电气与计算机工程系）
Hyperspectral image classification (HIC) is an active research topic in remote sensing. Hyperspectral images typically generate large data cubes posing big challenges in data acquisition, storage, transmission and processing. To overcome these limitations, this paper develops a novel deep learning HIC approach based on compressive measurements of coded-aperture snapshot spectral imagers (CASSI), without reconstructing the complete hyperspectral data cube. A new kind of deep learning strategy, namely 3D coded convolutional neural network (3D-CCNN) is proposed to efficiently solve for the classi…
In this paper, we study the problem of hyperspectral pixel classification based on the recently proposed architectures for compressive whisk-broom hyperspectral imagers without the need to reconstruct the complete data cube. A clear advantage of classification in the compressed domain is its suitability for real-time on-site processing of the sensed data. Moreover, it is assumed that the training process also takes place in the compressed domain, thus, isolating the classification unit from the recovery unit at the receiver's side. We show that, perhaps surprisingly, using distinct measurement…
UCLouvain（天主教鲁汶大学）；Instituto de Telecomunicações（葡萄牙电信研究所）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等理工学院）
Single Pixel (SP) imaging is now a reality in many applications, e.g., biomedical ultrathin endoscope and fluorescent spectroscopy. In this context, many schemes exist to improve the light throughput of these device, e.g., using structured illumination driven by compressive sensing theory. In this work, we consider the combination of SP imaging with Fourier Transform Interferometry (SP-FTI) to reach high-resolution HyperSpectral (HS) imaging, as desirable, e.g., in fluorescent spectroscopy. While this association is not new, we here focus on optimizing the spatial illumination, structured as H…
Universidad Industrial de Santander（桑坦德工业大学）；Stanford University（斯坦福大学）
Compressive imaging using coded apertures (CA) is a powerful technique that can be used to recover depth, light fields, hyperspectral images and other quantities from a single snapshot. The performance of compressive imaging systems based on CAs mostly depends on two factors: the properties of the mask's attenuation pattern, that we refer to as "codification" and the computational techniques used to recover the quantity of interest from the coded snapshot. In this work, we introduce the idea of using time-varying CAs synchronized with spatially varying pixel shutters. We divide the exposure of…
University of California, Berkeley（加州大学伯克利分校）；Orange Coast College（奥兰治海岸学院）
Compressive lensless imagers enable novel applications in an extremely compact device, requiring only a phase or amplitude mask placed close to the sensor. They have been demonstrated for 2D and 3D microscopy, single-shot video, and single-shot hyperspectral imaging; in each of these cases, a compressive-sensing-based inverse problem is solved in order to recover a 3D data-cube from a 2D measurement. Typically, this is accomplished using convex optimization and hand-picked priors. Alternatively, deep learning-based reconstruction methods offer the promise of better priors, but require many tho…
Santa Clara University（圣克拉拉大学）；ETH Zürich（苏黎世联邦理工学院）；Westlake University（西湖大学）；Northeastern University（东北大学）
The emerging technology of snapshot compressive imaging (SCI) enables capturing high dimensional (HD) data in an efficient way. It is generally implemented by two components: an optical encoder that compresses HD signals into a 2D measurement and an algorithm decoder that retrieves the HD data upon the hardware-encoded measurement. Over a broad range of SCI applications, hyperspectral imaging (HSI) and video compressive sensing have received significant research attention in recent years. Among existing SCI reconstruction algorithms, deep learning-based methods stand out as their promising per…
Beijing Institute of Technology（北京理工大学）；Beijing Normal University（北京师范大学）
Snapshot hyperspectral imaging can capture the 3D hyperspectral image (HSI) with a single 2D measurement and has attracted increasing attention recently. Recovering the underlying HSI from the compressive measurement is an ill-posed problem and exploiting the image prior is essential for solving this ill-posed problem. However, existing reconstruction methods always start from modeling image prior with the 1D vector or 2D matrix and cannot fully exploit the structurally spectral-spatial nature in 3D HSI, thus leading to a poor fidelity. In this paper, we propose an effective high-order tensor …
S^2-Transformer for Mask-Aware Hyperspectral Image Reconstruction
School of Information, Rochester Institute of Technology（罗切斯特理工学院信息学院）；Meta GenAI（元宇宙生成式人工智能部门）；Shanghai Jiao Tong University（上海交通大学）；Westlake University（西湖大学）
Spectrum from Defocus: Fast Spectral Imaging with Chromatic Focal Stack
Department of Computer Science, McCormick School of Engineering, Northwestern University（西北大学计算机科学系、麦科马克工程学院）；Dolby Laboratories, Inc.（杜比实验室公司）；Elmore Family School of Electrical and Computer Engineering, Purdue University（帕克大学电子与计算机工程学院）
Bell Labs（贝尔实验室）；College of Optical Sciences, University of Arizona（亚利桑那大学光学科学学院）；Northwestern University（西北大学）
Capturing high-dimensional (HD) data is a long-term challenge in signal processing and related fields. Snapshot compressive imaging (SCI) uses a two-dimensional (2D) detector to capture HD ($\ge3$D) data in a {\em snapshot} measurement. Via novel optical designs, the 2D detector samples the HD data in a {\em compressive} manner; following this, algorithms are employed to reconstruct the desired HD data-cube. SCI has been used in hyperspectral imaging, video, holography, tomography, focal depth imaging, polarization imaging, microscopy, \etc.~Though the hardware has been investigated for more t…
Beijing University of Posts and Telecommunications（北京邮电大学）；Westlake University（西湖大学）
We consider using {\bf\em untrained neural networks} to solve the reconstruction problem of snapshot compressive imaging (SCI), which uses a two-dimensional (2D) detector to capture a high-dimensional (usually 3D) data-cube in a compressed manner. Various SCI systems have been built in recent years to capture data such as high-speed videos, hyperspectral images, and the state-of-the-art reconstruction is obtained by the deep neural networks. However, most of these networks are trained in an end-to-end manner by a large amount of corpus with sometimes simulated ground truth, measurement pairs. …
In coded aperture snapshot spectral imaging (CASSI) system, the real-world hyperspectral image (HSI) can be reconstructed from the captured compressive image in a snapshot. Model-based HSI reconstruction methods employed hand-crafted priors to solve the reconstruction problem, but most of which achieved limited success due to the poor representation capability of these hand-crafted priors. Deep learning based methods learning the mappings between the compressive images and the HSIs directly achieved much better results. Yet, it is nontrivial to design a powerful deep network heuristically for …
Tsinghua University（清华大学）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）；University of Würzburg（维尔茨堡大学）
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradati…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zurich（苏黎世联邦理工学院）
The rapid development of deep learning provides a better solution for the end-to-end reconstruction of hyperspectral image (HSI). However, existing learning-based methods have two major defects. Firstly, networks with self-attention usually sacrifice internal resolution to balance model performance against complexity, losing fine-grained high-resolution (HR) features. Secondly, even if the optimization focusing on spatial-spectral domain learning (SDL) converges to the ideal solution, there is still a significant visual difference between the reconstructed HSI and the truth. Therefore, we prop…
Phy-CoSF: Physics-Guided Continuous Spectral Fields Reconstruction and Super-Resolution for Snapshot Compressive Imaging
College of Communication Engineering, Jilin University, Changchun 130012, China.；School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, China.；School of Electrical \& Electronic Engineering, Nanyang Technological University, Singapore 639798.；Department of Computer；Information Science, University of Macau, Macau 999078, China.；College of Computer Science；Technology, Jilin University, Changchun 130012, China.；College of Artificial Intelligence, Jilin University, Changchun 130012, China.；School of Information；Communication Engineering, University of Electronic Science
