A blind compressive sensing algorithm is proposed to reconstruct hyperspectral images from spectrally-compressed measurements.The wavelength-dependent data are coded and then superposed, mapping the three-dimensional hyperspectral datacube to a two-dimensional image. The inversion algorithm learns a dictionary {\em in situ} from the measurements via global-local shrinkage priors. By using RGB images as side information of the compressive sensing system, the proposed approach is extended to learn a coupled dictionary from the joint dataset of the compressed measurements and the corresponding RGB images, to improve reconstruction quality. A prototype camera is built using a liquid-crystal-on-silicon modulator. Experimental reconstructions of hyperspectral datacubes from both simulated and real compressed measurements demonstrate the efficacy of the proposed inversion algorithm, the feasibility of the camera and the benefit of side information.
Case Western Reserve University（凯斯西储大学）；Commonwealth Scientific and Industrial Research Organisation (CSIRO)（联邦科学与工业研究组织（CSIRO））；University of Queensland（昆士兰大学）；University of New South Wales（新南威尔士大学）
In this paper, we propose a new sampling strategy for hyperspectral signals that is based on dictionary learning and singular value decomposition (SVD). Specifically, we first learn a sparsifying dictionary from training spectral data using dictionary learning. We then perform an SVD on the dictionary and use the first few left singular vectors as the rows of the measurement matrix to obtain the compressive measurements for reconstruction. The proposed method provides significant improvement over the conventional compressive sensing approaches. The reconstruction performance is further improve…
UCLouvain（鲁汶大学）；Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）
This paper introduces a single-pixel HyperSpectral (HS) imaging framework based on Fourier Transform Interferometry (FTI). By combining a space-time coding of the light illumination with partial interferometric observations of a collimated light beam (observed by a single pixel), our system benefits from (i) reduced measurement rate and light-exposure of the observed object compared to common (Nyquist) FTI imagers, and (ii) high spectral resolution as desirable in, e.g., Fluorescence Spectroscopy (FS). From the principles of compressive sensing with multilevel sampling, our method leverages th…
Nanjing University of Information Science and Technology（南京信息工程大学）；National University of Singapore（新加坡国立大学）
Hyperspectral compressive imaging takes advantage of compressive sensing theory to achieve coded aperture snapshot measurement without temporal scanning, and the entire three-dimensional spatial-spectral data is captured by a two-dimensional projection during a single integration period. Its core issue is how to reconstruct the underlying hyperspectral image using compressive sensing reconstruction algorithms. Due to the diversity in the spectral response characteristics and wavelength range of different spectral imaging devices, previous works are often inadequate to capture complex spectral …
Tsinghua University（清华大学）；Nokia Bell Labs（诺基亚贝尔实验室）；Duke University（杜克大学）
Snapshot compressive imaging (SCI) refers to compressive imaging systems where multiple frames are mapped into a single measurement, with video compressive imaging and hyperspectral compressive imaging as two representative applications. Though exciting results of high-speed videos and hyperspectral images have been demonstrated, the poor reconstruction quality precludes SCI from wide applications.This paper aims to boost the reconstruction quality of SCI via exploiting the high-dimensional structure in the desired signal. We build a joint model to integrate the nonlocal self-similarity of vid…
Compressed Sensing (CS) is suitable for remote acquisition of hyperspectral images for earth observation, since it could exploit the strong spatial and spectral correlations, llowing to simplify the architecture of the onboard sensors. Solutions proposed so far tend to decouple spatial and spectral dimensions to reduce the complexity of the reconstruction, not taking into account that onboard sensors progressively acquire spectral rows rather than acquiring spectral channels. For this reason, we propose a novel progressive CS architecture based on separate sensing of spectral rows and joint re…
RIKEN（理化学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；University of Tokyo（东京大学）；Nokia Bell Labs（诺基亚贝尔实验室）
Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) using a single coded two-dimensional (2D) measurement, in which algorithms are used to perform the inverse problem. Due to the ill-posed nature, various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. Unfortunately, the accuracy and computational complexity are unsatisfied. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurement, in thi…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Hyperspectral imaging is an essential imaging modality for a wide range of applications, especially in remote sensing, agriculture, and medicine. Inspired by existing hyperspectral cameras that are either slow, expensive, or bulky, reconstructing hyperspectral images (HSIs) from a low-budget snapshot measurement has drawn wide attention. By mapping a truncated numerical optimization algorithm into a network with a fixed number of phases, recent deep unfolding networks (DUNs) for spectral snapshot compressive sensing (SCI) have achieved remarkable success. However, DUNs are far from reaching th…
Colorado State University（科罗拉多州立大学）；Physical Sciences Inc.（物理科学公司）
Compressive sensing (CS) is a method of sampling which permits some classes of signals to be reconstructed with high accuracy even when they were under-sampled. In this paper we explore a phenomenon in which bandwise CS sampling of a hyperspectral data cube followed by reconstruction can actually result in amplification of chemical signals contained in the cube. Perhaps most surprisingly, chemical signal amplification generally seems to increase as the level of sampling decreases. In some examples, the chemical signal is significantly stronger in a data cube reconstructed from 10% CS sampling …
University of Oxford（牛津大学）；Ludwig–Maximilians–Universität München（慕尼黑大学）；John Adams Institute for Accelerator Science（约翰·亚当斯加速器科学研究所）
Presented is a novel way to combine snapshot compressive imaging and lateral shearing interferometry in order to capture the spatio-spectral phase of an ultrashort laser pulse in a single shot. A deep unrolling algorithm is utilised for the snapshot compressive imaging reconstruction due to its parameter efficiency and superior speed relative to other methods, potentially allowing for online reconstruction. The algorithm's regularisation term is represented using neural network with 3D convolutional layers, to exploit the spatio-spectral correlations that exist in laser wavefronts. Compressed …
Dual Hyperspectral Mamba for Efficient Spectral Compressive Imaging
The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Hunan University（湖南大学）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tsinghua University（清华大学）；Peking University（北京大学）
Hyperspectral imaging plays a pivotal role in a wide range of applications, like remote sensing, medicine, and cytology. By acquiring 3D hyperspectral images (HSIs) via 2D sensors, the coded aperture snapshot spectral imaging (CASSI) has achieved great success due to its hardware-friendly implementation and fast imaging speed. However, for some less spectrally sparse scenes, single snapshot and unreasonable coded aperture design tend to make HSI recovery more ill-posed and yield poor spatial and spectral fidelity. In this paper, we propose a novel Progressive Content-Aware CASSI framework, dub…
Key Laboratory of Photoelectronic Imaging Technology and System of Ministry of Education of China, School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院光电成像技术与系统教育部重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（特拉华大学电气与计算机工程系）
Hyperspectral image classification (HIC) is an active research topic in remote sensing. Hyperspectral images typically generate large data cubes posing big challenges in data acquisition, storage, transmission and processing. To overcome these limitations, this paper develops a novel deep learning HIC approach based on compressive measurements of coded-aperture snapshot spectral imagers (CASSI), without reconstructing the complete hyperspectral data cube. A new kind of deep learning strategy, namely 3D coded convolutional neural network (3D-CCNN) is proposed to efficiently solve for the classi…
In this paper, we study the problem of hyperspectral pixel classification based on the recently proposed architectures for compressive whisk-broom hyperspectral imagers without the need to reconstruct the complete data cube. A clear advantage of classification in the compressed domain is its suitability for real-time on-site processing of the sensed data. Moreover, it is assumed that the training process also takes place in the compressed domain, thus, isolating the classification unit from the recovery unit at the receiver's side. We show that, perhaps surprisingly, using distinct measurement…
UCLouvain（天主教鲁汶大学）；Instituto de Telecomunicações（葡萄牙电信研究所）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等理工学院）
Single Pixel (SP) imaging is now a reality in many applications, e.g., biomedical ultrathin endoscope and fluorescent spectroscopy. In this context, many schemes exist to improve the light throughput of these device, e.g., using structured illumination driven by compressive sensing theory. In this work, we consider the combination of SP imaging with Fourier Transform Interferometry (SP-FTI) to reach high-resolution HyperSpectral (HS) imaging, as desirable, e.g., in fluorescent spectroscopy. While this association is not new, we here focus on optimizing the spatial illumination, structured as H…
HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University of Iceland（冰岛大学）；Griffith University（格里菲斯大学）
Universidad Industrial de Santander（桑坦德工业大学）；Stanford University（斯坦福大学）
Compressive imaging using coded apertures (CA) is a powerful technique that can be used to recover depth, light fields, hyperspectral images and other quantities from a single snapshot. The performance of compressive imaging systems based on CAs mostly depends on two factors: the properties of the mask's attenuation pattern, that we refer to as "codification" and the computational techniques used to recover the quantity of interest from the coded snapshot. In this work, we introduce the idea of using time-varying CAs synchronized with spatially varying pixel shutters. We divide the exposure of…
University of California, Berkeley（加州大学伯克利分校）；Orange Coast College（奥兰治海岸学院）
Compressive lensless imagers enable novel applications in an extremely compact device, requiring only a phase or amplitude mask placed close to the sensor. They have been demonstrated for 2D and 3D microscopy, single-shot video, and single-shot hyperspectral imaging; in each of these cases, a compressive-sensing-based inverse problem is solved in order to recover a 3D data-cube from a 2D measurement. Typically, this is accomplished using convex optimization and hand-picked priors. Alternatively, deep learning-based reconstruction methods offer the promise of better priors, but require many tho…
Tampere University（坦佩雷大学）；Aarhus University（奥胡斯大学）
Compressive Learning is an emerging topic that combines signal acquisition via compressive sensing and machine learning to perform inference tasks directly on a small number of measurements. Many data modalities naturally have a multi-dimensional or tensorial format, with each dimension or tensor mode representing different features such as the spatial and temporal information in video sequences or the spatial and spectral information in hyperspectral images. However, in existing compressive learning frameworks, the compressive sensing component utilizes either random or learned linear project…
Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）
Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) betw…
Santa Clara University（圣克拉拉大学）；ETH Zürich（苏黎世联邦理工学院）；Westlake University（西湖大学）；Northeastern University（东北大学）
The emerging technology of snapshot compressive imaging (SCI) enables capturing high dimensional (HD) data in an efficient way. It is generally implemented by two components: an optical encoder that compresses HD signals into a 2D measurement and an algorithm decoder that retrieves the HD data upon the hardware-encoded measurement. Over a broad range of SCI applications, hyperspectral imaging (HSI) and video compressive sensing have received significant research attention in recent years. Among existing SCI reconstruction algorithms, deep learning-based methods stand out as their promising per…
Beijing Institute of Technology（北京理工大学）；Beijing Normal University（北京师范大学）
Snapshot hyperspectral imaging can capture the 3D hyperspectral image (HSI) with a single 2D measurement and has attracted increasing attention recently. Recovering the underlying HSI from the compressive measurement is an ill-posed problem and exploiting the image prior is essential for solving this ill-posed problem. However, existing reconstruction methods always start from modeling image prior with the 1D vector or 2D matrix and cannot fully exploit the structurally spectral-spatial nature in 3D HSI, thus leading to a poor fidelity. In this paper, we propose an effective high-order tensor …
This work addresses the problem of extracting deeply learned features directly from compressive measurements. There has been no work in this area. Existing deep learning tools only give good results when applied on the full signal, that too usually after preprocessing. These techniques require the signal to be reconstructed first. In this work we show that by learning directly from the compressed domain, considerably better results can be obtained. This work extends the recently proposed framework of deep matrix factorization in combination with blind compressed sensing; hence the term deep bl…
S^2-Transformer for Mask-Aware Hyperspectral Image Reconstruction
School of Information, Rochester Institute of Technology（罗切斯特理工学院信息学院）；Meta GenAI（元宇宙生成式人工智能部门）；Shanghai Jiao Tong University（上海交通大学）；Westlake University（西湖大学）
Spectrum from Defocus: Fast Spectral Imaging with Chromatic Focal Stack
Department of Computer Science, McCormick School of Engineering, Northwestern University（西北大学计算机科学系、麦科马克工程学院）；Dolby Laboratories, Inc.（杜比实验室公司）；Elmore Family School of Electrical and Computer Engineering, Purdue University（帕克大学电子与计算机工程学院）
Bell Labs（贝尔实验室）；College of Optical Sciences, University of Arizona（亚利桑那大学光学科学学院）；Northwestern University（西北大学）
Capturing high-dimensional (HD) data is a long-term challenge in signal processing and related fields. Snapshot compressive imaging (SCI) uses a two-dimensional (2D) detector to capture HD ($\ge3$D) data in a {\em snapshot} measurement. Via novel optical designs, the 2D detector samples the HD data in a {\em compressive} manner; following this, algorithms are employed to reconstruct the desired HD data-cube. SCI has been used in hyperspectral imaging, video, holography, tomography, focal depth imaging, polarization imaging, microscopy, \etc.~Though the hardware has been investigated for more t…
Beijing University of Posts and Telecommunications（北京邮电大学）；Westlake University（西湖大学）
We consider using {\bf\em untrained neural networks} to solve the reconstruction problem of snapshot compressive imaging (SCI), which uses a two-dimensional (2D) detector to capture a high-dimensional (usually 3D) data-cube in a compressed manner. Various SCI systems have been built in recent years to capture data such as high-speed videos, hyperspectral images, and the state-of-the-art reconstruction is obtained by the deep neural networks. However, most of these networks are trained in an end-to-end manner by a large amount of corpus with sometimes simulated ground truth, measurement pairs. …
In coded aperture snapshot spectral imaging (CASSI) system, the real-world hyperspectral image (HSI) can be reconstructed from the captured compressive image in a snapshot. Model-based HSI reconstruction methods employed hand-crafted priors to solve the reconstruction problem, but most of which achieved limited success due to the poor representation capability of these hand-crafted priors. Deep learning based methods learning the mappings between the compressive images and the HSIs directly achieved much better results. Yet, it is nontrivial to design a powerful deep network heuristically for …
Universidad Rey Juan Carlos（胡安卡洛斯国王大学）；Universidad Industrial de Santander（桑坦德工业大学）
Image fusion aims at estimating a high-resolution spectral image from a low-spatial-resolution hyperspectral image and a low-spectral-resolution multispectral image. In this regard, compressive spectral imaging (CSI) has emerged as an acquisition framework that captures the relevant information of spectral images using a reduced number of measurements. Recently, various image fusion methods from CSI measurements have been proposed. However, these methods exhibit high running times and face the challenging task of choosing sparsity-inducing bases. In this paper, a deep network under the algorit…
Tsinghua University（清华大学）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）；University of Würzburg（维尔茨堡大学）
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradati…
City University of Hong Kong（香港城市大学）；Caritas Institute of Higher Education（明爱专上学院）
This paper tackles the challenging problem of hyperspectral (HS) image denoising. Unlike existing deep learning-based methods usually adopting complicated network architectures or empirically stacking off-the-shelf modules to pursue performance improvement, we focus on the efficient and effective feature extraction manner for capturing the high-dimensional characteristics of HS images. To be specific, based on the theoretical analysis that increasing the rank of the matrix formed by the unfolded convolutional kernels can promote feature diversity, we propose rank-enhanced low-dimensional convo…
In a plethora of applications dealing with inverse problems, e.g. in image processing, social networks, compressive sensing, biological data processing etc., the signal of interest is known to be structured in several ways at the same time. This premise has recently guided the research to the innovative and meaningful idea of imposing multiple constraints on the parameters involved in the problem under study. For instance, when dealing with problems whose parameters form sparse and low-rank matrices, the adoption of suitably combined constraints imposing sparsity and low-rankness, is expected …
BITS Pilani K K Birla Goa Campus（比尔拉科学技术学院K·K·比拉尔果阿校区）；APPCAIR centre（APPCAIR中心）
Hyperspectral image (HSI) classification is a topic of active research. One of the main challenges of HSI classification is the lack of reliable labelled samples. Various semi-supervised and unsupervised classification methods are proposed to handle the low number of labelled samples. Chief among them are graph convolution networks (GCN) and their variants. These approaches exploit the graph structure for semi-supervised and unsupervised classification. While several of these methods implicitly construct edge-weights, to our knowledge, not much work has been done to estimate the edge-weights e…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zurich（苏黎世联邦理工学院）
The rapid development of deep learning provides a better solution for the end-to-end reconstruction of hyperspectral image (HSI). However, existing learning-based methods have two major defects. Firstly, networks with self-attention usually sacrifice internal resolution to balance model performance against complexity, losing fine-grained high-resolution (HR) features. Secondly, even if the optimization focusing on spatial-spectral domain learning (SDL) converges to the ideal solution, there is still a significant visual difference between the reconstructed HSI and the truth. Therefore, we prop…
We propose a new method, {\it robust binary fused compressive sensing} (RoBFCS), to recover sparse piece-wise smooth signals from 1-bit compressive measurements. The proposed method is a modification of our previous {\it binary fused compressive sensing} (BFCS) algorithm, which is based on the {\it binary iterative hard thresholding} (BIHT) algorithm. As in BIHT, the data term of the objective function is a one-sided $\ell_1$ (or $\ell_2$) norm. Experiments show that the proposed algorithm is able to take advantage of the piece-wise smoothness of the original signal and detect sign flips and c…
Phy-CoSF: Physics-Guided Continuous Spectral Fields Reconstruction and Super-Resolution for Snapshot Compressive Imaging
College of Communication Engineering, Jilin University, Changchun 130012, China.；School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, China.；School of Electrical \& Electronic Engineering, Nanyang Technological University, Singapore 639798.；Department of Computer；Information Science, University of Macau, Macau 999078, China.；College of Computer Science；Technology, Jilin University, Changchun 130012, China.；College of Artificial Intelligence, Jilin University, Changchun 130012, China.；School of Information；Communication Engineering, University of Electronic Science
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico, Technical University of Lisbon（里斯本技术大学高等技术学院）
We propose a new method, {\it binary fused compressive sensing} (BFCS), to recover sparse piece-wise smooth signals from 1-bit compressive measurements. The proposed algorithm is a modification of the previous {\it binary iterative hard thresholding} (BIHT) algorithm, where, in addition to the sparsity constraint, the total-variation of the recovered signal is upper constrained. As in BIHT, the data term of the objective function is an one-sided $\ell_1$ (or $\ell_2$) norm. Experiments on the recovery of sparse piece-wise smooth signals show that the proposed algorithm is able to take advantag…
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）
We propose a new approach, {\it two-dimensional fused binary compressive sensing} (2DFBCS) to recover 2D sparse piece-wise signals from 1-bit measurements, exploiting 2D group sparsity for 1-bit compressive sensing recovery. The proposed method is a modified 2D version of the previous {\it binary iterative hard thresholding} (2DBIHT) algorithm, where the objective function includes a 2D one-sided $\ell_1$ (or $\ell_2$) penalty function encouraging agreement with the observed data, an indicator function of $K$-sparsity, and a total variation (TV) or modified TV (MTV) constraint. The subgradient…
Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat
National Cheng Kung University（成功大学）；Institute of Data Science（数据科学研究所）；Miin Wu School of Computing（米润计算机学院）；Department of Statistics（统计学系）；Department of Computer Science（计算机科学系）
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Technical University of Denmark（丹麦技术大学）
This work considers reconstructing a target signal in a context of distributed sparse sources. We propose an efficient reconstruction algorithm with the aid of other given sources as multiple side information (SI). The proposed algorithm takes advantage of compressive sensing (CS) with SI and adaptive weights by solving a proposed weighted $n$-$\ell_{1}$ minimization. The proposed algorithm computes the adaptive weights in two levels, first each individual intra-SI and then inter-SI weights are iteratively updated at every reconstructed iteration. This two-level optimization leads the proposed…
Hyperspectral/multispectral imaging (HSI/MSI) contains rich information clinical applications, such as 1) narrow band imaging for vascular visualisation; 2) oxygen saturation for intraoperative perfusion monitoring and clinical decision making [1]; 3) tissue classification and identification of pathology [2]. The current systems which provide pixel-level HSI/MSI signal can be generally divided into two types: spatial scanning and spectral scanning. However, the trade-off between spatial/spectral resolution, the acquisition time, and the hardware complexity hampers implementation in real-world …
