In the coded aperture snapshot spectral imaging system, Deep Unfolding Networks (DUNs) have made impressive progress in recovering 3D hyperspectral images (HSIs) from a single 2D measurement. However, the inherent nonlinear and ill-posed characteristics of HSI reconstruction still pose challenges to existing methods in terms of accuracy and stability. To address this issue, we propose a Mamba-inspired Joint Unfolding Network (MiJUN), which integrates physics-embedded DUNs with learning-based HSI imaging. Firstly, leveraging the concept of trapezoid discretization to expand the representation space of unfolding networks, we introduce an accelerated unfolding network scheme. This approach can be interpreted as a generalized accelerated half-quadratic splitting with a second-order differential equation, which reduces the reliance on initial optimization stages and addresses challenges related to long-range interactions. Crucially, within the Mamba framework, we restructure the Mamba-inspired global-to-local attention mechanism by incorporating a selective state space model and an attention mechanism. This effectively reinterprets Mamba as a variant of the Transformer} architecture, improving its adaptability and efficiency. Furthermore, we refine the scanning strategy with Mamba by integrating the tensor mode-$k$ unfolding into the Mamba network. This approach emphasizes the low-rank properties of tensors along various modes, while conveniently facilitating 12 scanning directions. Numerical and visual comparisons on both simulation and real datasets demonstrate the superiority of our proposed MiJUN, and achieving overwhelming detail representation.
Variational Deep Unfolding with Mamba-Based Nonlocal Modeling for Underwater Image Enhancement
Institute of Applied Computing and Community Code (IAC3)（应用计算与社区代码研究所 (IAC3)）；Dept. of Mathematics and Computer Science, Universitat de les Illes Balears（巴利阿里群岛大学数学与计算机科学系）
Deep Probabilistic Unfolding for Quantized Compressive Sensing
Westlake University, School of Engineering, Hangzhou, Zhejiang, China（西湖大学工程学院，杭州，浙江，中国）；Vivo Mobile Communication Co., Ltd., Hangzhou, Zhejiang, China（Vivo移动通信有限公司，杭州，浙江，中国）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）；Peking University（北京大学）
Recent deep network-based compressive sensing (CS) methods have achieved great success. However, most of them regard different sampling matrices as different independent tasks and need to train a specific model for each target sampling matrix. Such practices give rise to inefficiency in computing and suffer from poor generalization ability. In this paper, we propose a novel COntrollable Arbitrary-Sampling neTwork, dubbed COAST, to solve CS problems of arbitrary-sampling matrices (including unseen sampling matrices) with one single model. Under the optimization-inspired deep unfolding framework…
Recursions Are All You Need: Towards Efficient Deep Unfolding Networks
King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（沙特数据与人工智能局-法赫德国王石油与矿业大学人工智能联合研究中心）
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Jiao Tong University（上海交通大学）
Most compressive sensing (CS) reconstruction methods can be divided into two categories, i.e. model-based methods and classical deep network methods. By unfolding the iterative optimization algorithm for model-based methods onto networks, deep unfolding methods have the good interpretation of model-based methods and the high speed of classical deep network methods. In this paper, to solve the visual image CS problem, we propose a deep unfolding model dubbed AMP-Net. Rather than learning regularization terms, it is established by unfolding the iterative denoising process of the well-known appro…
SlowFast-SCI: Slow-Fast Deep Unfolding Learning for Spectral Compressive Imaging
Harvard University（哈佛大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Shanghai Jiaotong University（上海交通大学）；City University of Hong Kong（香港城市大学）；Harbin Institute of Technology, Weihai（哈尔滨工业大学（威海））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Tsinghua University（清华大学）；Shenzhen Institute of Future Media Technology（深圳未来媒体技术研究院）；Westlake University（西湖大学）；ETH Zürich（苏黎世联邦理工学院）；University of Würzburg（维尔茨堡大学）
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradati…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）
While deep neural networks have achieved impressive success in image compressive sensing (CS), most of them lack flexibility when dealing with multi-ratio tasks and multi-scene images in practical applications. To tackle these challenges, we propose a novel end-to-end flexible ISTA-unfolding deep network, dubbed ISTA-Net++, with superior performance and strong flexibility. Specifically, by developing a dynamic unfolding strategy, our model enjoys the adaptability of handling CS problems with different ratios, i.e., multi-ratio tasks, through a single model. A cross-block strategy is further ut…
Computer Network Information Center, Chinese Academy of Science（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；Westlake University（西湖大学）
We consider the reconstruction problem of video compressive sensing (VCS) under the deep unfolding/rolling structure. Yet, we aim to build a flexible and concise model using minimum stages. Different from existing deep unfolding networks used for inverse problems, where more stages are used for higher performance but without flexibility to different masks and scales, hereby we show that a 2-stage deep unfolding network can lead to the state-of-the-art (SOTA) results (with a 1.7dB gain in PSNR over the single stage model, RevSCI) in VCS. The proposed method possesses the properties of adaptatio…
University of Electronic Science and Technology of China（电子科技大学）；University of Copenhagen（哥本哈根大学）
Deep unfolding networks (DUNs) have proven to be a viable approach to compressive sensing (CS). In this work, we propose a DUN called low-rank CS network (LR-CSNet) for natural image CS. Real-world image patches are often well-represented by low-rank approximations. LR-CSNet exploits this property by adding a low-rank prior to the CS optimization task. We derive a corresponding iterative optimization procedure using variable splitting, which is then translated to a new DUN architecture. The architecture uses low-rank generation modules (LRGMs), which learn low-rank matrix factorizations, as we…
Latent Diffusion Prior Enhanced Deep Unfolding for Snapshot Spectral Compressive Imaging
Zhejiang University（浙江大学）；Westlake University（西湖大学）；School of Engineering, Westlake University（西湖大学工程学院）；Xidian University（西安电子科技大学）；School of Cyber Engineering, Xidian University（西安电子科技大学网络与工程学院）；Beijing Institute of Technology（北京理工大学）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Snapshot compressive imaging (SCI) aims to record three-dimensional signals via a two-dimensional camera. For the sake of building a fast and accurate SCI recovery algorithm, we incorporate the interpretability of model-based methods and the speed of learning-based ones and present a novel dense deep unfolding network (DUN) with 3D-CNN prior for SCI, where each phase is unrolled from an iteration of Half-Quadratic Splitting (HQS). To better exploit the spatial-temporal correlation among frames and address the problem of information loss between adjacent phases in existing DUNs, we propose to a…
Mapping a truncated optimization method into a deep neural network, deep unfolding network (DUN) has attracted growing attention in compressive sensing (CS) due to its good interpretability and high performance. Each stage in DUNs corresponds to one iteration in optimization. By understanding DUNs from the perspective of the human brain's memory processing, we find there exists two issues in existing DUNs. One is the information between every two adjacent stages, which can be regarded as short-term memory, is usually lost seriously. The other is no explicit mechanism to ensure that the previou…
Dual Hyperspectral Mamba for Efficient Spectral Compressive Imaging
The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Hunan University（湖南大学）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Hyperspectral imaging is an essential imaging modality for a wide range of applications, especially in remote sensing, agriculture, and medicine. Inspired by existing hyperspectral cameras that are either slow, expensive, or bulky, reconstructing hyperspectral images (HSIs) from a low-budget snapshot measurement has drawn wide attention. By mapping a truncated numerical optimization algorithm into a network with a fixed number of phases, recent deep unfolding networks (DUNs) for spectral snapshot compressive sensing (SCI) have achieved remarkable success. However, DUNs are far from reaching th…
Phy-CoSF: Physics-Guided Continuous Spectral Fields Reconstruction and Super-Resolution for Snapshot Compressive Imaging
College of Communication Engineering, Jilin University, Changchun 130012, China.；School of Engineering, Westlake University, Hangzhou, Zhejiang 310024, China.；School of Electrical \& Electronic Engineering, Nanyang Technological University, Singapore 639798.；Department of Computer；Information Science, University of Macau, Macau 999078, China.；College of Computer Science；Technology, Jilin University, Changchun 130012, China.；College of Artificial Intelligence, Jilin University, Changchun 130012, China.；School of Information；Communication Engineering, University of Electronic Science
Cross-Scan Mamba with Masked Training for Robust Spectral Imaging
University of Helsinki（赫尔辛基大学）；IMEC（微电子中心）；Universiteit Gent（根特大学）；Northwestern Polytechnical University（西北工业大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
D3C2-Net: Dual-Domain Deep Convolutional Coding Network for Compressive Sensing
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Graduate School at Shenzhen, Tsinghua University（清华大学深圳研究生院）；ByteDance Inc（字节跳动公司）；School of Software Engineering, Tongji University（同济大学软件学院）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）
Deep LoRA-Unfolding Networks for Image Restoration
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（计算机科学与技术学院，哈尔滨工业大学（深圳））；Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（图像通信与网络工程院，上海交通大学）；School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen)（电子与信息工程学院，哈尔滨工业大学（深圳））
University of Electronic Science and Technology of China（电子科技大学）
Deep learning (DL)-based tomographic SAR imaging algorithms are gradually being studied. Typically, they use an unfolding network to mimic the iterative calculation of the classical compressive sensing (CS)-based methods and process each range-azimuth unit individually. However, only one-dimensional features are effectively utilized in this way. The correlation between adjacent resolution units is ignored directly. To address that, we propose a new model-data-driven network to achieve tomoSAR imaging based on multi-dimensional features. Guided by the deep unfolding methodology, a two-dimension…
It is promising to solve linear inverse problems by unfolding iterative algorithms (e.g., iterative shrinkage thresholding algorithm (ISTA)) as deep neural networks (DNNs) with learnable parameters. However, existing ISTA-based unfolded algorithms restrict the network architectures for iterative updates with the partial weight coupling structure to guarantee convergence. In this paper, we propose hybrid ISTA to unfold ISTA with both pre-computed and learned parameters by incorporating free-form DNNs (i.e., DNNs with arbitrary feasible and reasonable network architectures), while ensuring theor…
RadioDUN: A Physics-Inspired Deep Unfolding Network for Radio Map Estimation
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；Pengcheng Laboratory（鹏城实验室）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）
