Hyperspectral images (HSIs) are often corrupted by a mixture of several types of noise during the acquisition process, e.g., Gaussian noise, impulse noise, dead lines, stripes, and many others. Such complex noise could degrade the quality of the acquired HSIs, limiting the precision of the subsequent processing. In this paper, we present a novel tensor-based HSI restoration approach by fully identifying the intrinsic structures of the clean HSI part and the mixed noise part respectively. Specifically, for the clean HSI part, we use tensor Tucker decomposition to describe the global correlation among all bands, and an anisotropic spatial-spectral total variation (SSTV) regularization to characterize the piecewise smooth structure in both spatial and spectral domains. For the mixed noise part, we adopt the $\ell_1$ norm regularization to detect the sparse noise, including stripes, impulse noise, and dead pixels. Despite that TV regulariztion has the ability of removing Gaussian noise, the Frobenius norm term is further used to model heavy Gaussian noise for some real-world scenarios. Then, we develop an efficient algorithm for solving the resulting optimization problem by using the augmented Lagrange multiplier (ALM) method. Finally, extensive experiments on simulated and real-world noise HSIs are carried out to demonstrate the superiority of the proposed method over the existing state-of-the-art ones.
Supervised learning with a convolutional neural network is recognized as a powerful means of image restoration. However, most such methods have been designed for application to grayscale and/or color images; therefore, they have limited success when applied to hyperspectral image restoration. This is partially owing to large datasets being difficult to collect, and also the heavy computational load associated with the restoration of an image with many spectral bands. To address this difficulty, we propose a novel self-supervised learning strategy for application to hyperspectral image restorat…
National Laboratory of Radar Signal Processing, Xidian University（西安电子科技大学雷达信号处理国家重点实验室）；Science and Technology on Multispectral Information Processing Laboratory, School of Automation, Huazhong University of Science and Technology（华中科技大学自动化学院多谱信息处理技术国家级重点实验室）
Hyperspectral imaging, providing abundant spatial and spectral information simultaneously, has attracted a lot of interest in recent years. Unfortunately, due to the hardware limitations, the hyperspectral image (HSI) is vulnerable to various degradations, such noises (random noise, HSI denoising), blurs (Gaussian and uniform blur, HSI deblurring), and down-sampled (both spectral and spatial downsample, HSI super-resolution). Previous HSI restoration methods are designed for one specific task only. Besides, most of them start from the 1-D vector or 2-D matrix models and cannot fully exploit th…
HIR-ALIGN: Enhancing Hyperspectral Image Restoration via Diffusion-Based Data Generation
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Mathematics and Statistics and the Ministry of Education Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学数学与统计学学院和教育部智能网络与网络安全重点实验室）；Pazhou Laboratory (Huangpu), Guangzhou（广州黄埔 Pazhou 实验室）
Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth "clean" hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where d…
Northwest A\&F University, College of Science, Yangling 712100, P.R. China
Several bandwise total variation (TV) regularized low-rank (LR)-based models have been proposed to remove mixed noise in hyperspectral images (HSIs). Conventionally, the rank of LR matrix is approximated using nuclear norm (NN). The NN is defined by adding all singular values together, which is essentially a $L_1$-norm of the singular values. It results in non-negligible approximation errors and thus the resulting matrix estimator can be significantly biased. Moreover, these bandwise TV-based methods exploit the spatial information in a separate manner. To cope with these problems, we propose …
PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation
Institute of Data Science, National Cheng Kung University（成功大学数据科学研究所）；Graduate Institute of Communication Engineering, National Taiwan University（台湾大学通讯工程学系）；NVIDIA（英伟达公司）
Wuhan University（武汉大学）；JD Explore Academy（京东探索研究院）；University of Sydney（悉尼大学）
Diffusion models have recently received a surge of interest due to their impressive performance for image restoration, especially in terms of noise robustness. However, existing diffusion-based methods are trained on a large amount of training data and perform very well in-distribution, but can be quite susceptible to distribution shift. This is especially inappropriate for data-starved hyperspectral image (HSI) restoration. To tackle this problem, this work puts forth a self-supervised diffusion model for HSI restoration, namely Denoising Diffusion Spatio-Spectral Model (\texttt{DDS2M}), whic…
RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Paradigm Inc.（第四范式有限公司）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；LISMARS, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) restoration, which includes the tasks of denoising, compressed HSI reconstruction and inpainting. Unfortunately, while its restoration performance benefits from more spectral bands, its runtime also substantially increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch group should lie in this global low-rank subspace. This motivates us to propose a unified paradigm combining the spatial and …
In the past decade, sparsity-driven regularization has led to significant improvements in image reconstruction. Traditional regularizers, such as total variation (TV), rely on analytical models of sparsity. However, increasingly the field is moving towards trainable models, inspired from deep learning. Deep image prior (DIP) is a recent regularization framework that uses a convolutional neural network (CNN) architecture without data-driven training. This paper extends the DIP framework by combining it with the traditional TV regularization. We show that the inclusion of TV leads to considerabl…
College of Medicine and Biological Information Engineering, Northeastern University（东北大学医学与生物信息工程学院）；Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education（教育部医学影像智能计算重点实验室）；Department of Electrical and Computer Engineering, Stevens Institute of Technology（史蒂文斯理工学院电气与计算机工程系）
The iterative weighted shrinkage-thresholding algorithm (IWSTA) has shown superiority to the classic unweighted iterative shrinkage-thresholding algorithm (ISTA) for solving linear inverse problems, which address the attributes differently. This paper proposes a new entropy regularized IWSTA (ERIWSTA) that adds an entropy regularizer to the cost function to measure the uncertainty of the weights to stimulate attributes to participate in problem solving. Then, the weights are solved with a Lagrange multiplier method to obtain a simple iterative update. The weights can be explained as the probab…
Deep-learning-based hyperspectral image (HSI) restoration methods have gained great popularity for their remarkable performance but often demand expensive network retraining whenever the specifics of task changes. In this paper, we propose to restore HSIs in a unified approach with an effective plug-and-play method, which can jointly retain the flexibility of optimization-based methods and utilize the powerful representation capability of deep neural networks. Specifically, we first develop a new deep HSI denoiser leveraging gated recurrent convolution units, short- and long-term skip connecti…
HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）
Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China（北京理工大学计算机科学与技术学院）；School of Computer Science, Wuhan University, Wuhan, Hubei, China（武汉大学计算机学院）；Zhongguancun Academy, Beijing, China（中关村学院）
Hyperspectral image (HSI) denoising aims to restore clean HSI from the noise-contaminated one. Noise contamination can often be caused during data acquisition and conversion. In this paper, we propose a novel spatial-spectral total variation (SSTV) regularized nonconvex local low-rank (LR) tensor approximation method to remove mixed noise in HSIs. From one aspect, the clean HSI data have its underlying local LR tensor property, even though the real HSI data may not be globally low-rank due to out-liers and non-Gaussian noise. According to this fact, we propose a novel tensor $L_γ$-norm to form…
The Norwegian Colour and Visual Computing Laboratory（挪威色彩与视觉计算实验室）；NTNU（挪威科技大学）
Deep learning algorithms have demonstrated state-of-the-art performance in various tasks of image restoration. This was made possible through the ability of CNNs to learn from large exemplar sets. However, the latter becomes an issue for hyperspectral image processing where datasets commonly consist of just a few images. In this work, we propose a new approach to denoising, inpainting, and super-resolution of hyperspectral image data using intrinsic properties of a CNN without any training. The performance of the given algorithm is shown to be comparable to the performance of trained networks,…
In this paper, we propose an alternating directional 3D quasi-recurrent neural network for hyperspectral image (HSI) denoising, which can effectively embed the domain knowledge -- structural spatio-spectral correlation and global correlation along spectrum. Specifically, 3D convolution is utilized to extract structural spatio-spectral correlation in an HSI, while a quasi-recurrent pooling function is employed to capture the global correlation along spectrum. Moreover, alternating directional structure is introduced to eliminate the causal dependency with no additional computation cost. The pro…
Hyperspectral image (HSI) denoising is a crucial preprocessing procedure to improve the performance of the subsequent HSI interpretation and applications. In this paper, a novel deep learning-based method for this task is proposed, by learning a non-linear end-to-end mapping between the noisy and clean HSIs with a combined spatial-spectral deep convolutional neural network (HSID-CNN). Both the spatial and spectral information are simultaneously assigned to the proposed network. In addition, multi-scale feature extraction and multi-level feature representation are respectively employed to captu…
Instituto de Telecomunicações（电信研究院）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）
This paper introduces two very fast and competitive hyperspectral image (HSI) restoration algorithms: fast hyperspectral denoising (FastHyDe), a denoising algorithm able to cope with Gaussian and Poissonian noise, and fast hyperspectral inpainting (FastHyIn), an inpainting algorithm to restore HSIs where some observations from known pixels in some known bands are missing. FastHyDe and FastHyIn fully exploit extremely compact and sparse HSI representations linked with their low-rank and self-similarity characteristics. In a series of experiments with simulated and real data, the newly introduce…
University at Albany, SUNY（纽约州立大学奥尔巴尼分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Several important classes of images such as text, barcode and pattern images have the property that pixels can only take a distinct subset of values. This knowledge can benefit the restoration of such images, but it has not been widely considered in current restoration methods. In this work, we describe an effective and efficient approach to incorporate the knowledge of distinct pixel values of the pristine images into the general regularized least squares restoration framework. We introduce a new regularizer that attains zero at the designated pixel values and becomes a quadratic penalty func…
College of Arts and Sciences, National University of Defense Technology（国防科技大学文理学院）
It has been an important approach of using matrix completion to perform image restoration. Most previous works on matrix completion focus on the low-rank property by imposing explicit constraints on the recovered matrix, such as the constraint of the nuclear norm or limiting the dimension of the matrix factorization component. Recently, theoretical works suggest that deep linear neural network has an implicit bias towards low rank on matrix completion. However, low rank is not adequate to reflect the intrinsic characteristics of a natural image. Thus, algorithms with only the constraint of low…
Qingdao University（青岛大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Kentucky（肯塔基大学）
It is a challenging task to remove heavy and mixed types of noise from Hyperspectral images (HSIs). In this paper, we propose a novel nonconvex approach to RPCA for HSI denoising, which adopts the log-determinant rank approximation and a novel $\ell_{2,\log}$ norm, to restrict the low-rank or column-wise sparse properties for the component matrices, respectively.For the $\ell_{2,\log}$-regularized shrinkage problem, we develop an efficient, closed-form solution, which is named $\ell_{2,\log}$-shrinkage operator, which can be generally used in other problems. Extensive experiments on both simul…
Deep learning based methods have achieved remarkable success in image restoration and enhancement, but most such methods rely on RGB input images. These methods fail to take into account the rich spectral distribution of natural images. We propose a deep architecture, SpecNet, which computes spectral profile to estimate pixel-wise dynamic range adjustment of a given image. First, we employ an unpaired cycle-consistent framework to generate hyperspectral images (HSI) from low-light input images. HSI is further used to generate a normal light image of the same scene. We incorporate a self-superv…
Spectral unmixing is an important tool in hyperspectral data analysis for estimating endmembers and abundance fractions in a mixed pixel. This paper examines the applicability of a recently developed algorithm called graph regularized nonnegative matrix factorization (GNMF) for this aim. The proposed approach exploits the intrinsic geometrical structure of the data besides considering positivity and full additivity constraints. Simulated data based on the measured spectral signatures, is used for evaluating the proposed algorithm. Results in terms of abundance angle distance (AAD) and spectral…
Recently, the low-rank property of different components extracted from the image has been considered in man hyperspectral image denoising methods. However, these methods usually unfold the 3D tensor to 2D matrix or 1D vector to exploit the prior information, such as nonlocal spatial self-similarity (NSS) and global spectral correlation (GSC), which break the intrinsic structure correlation of hyperspectral image (HSI) and thus lead to poor restoration quality. In addition, most of them suffer from heavy computational burden issues due to the involvement of singular value decomposition operatio…
Hyperspectral super-resolution (HSR) fuses a low-resolution hyperspectral image (HSI) and a high-resolution multispectral image (MSI) to obtain a high-resolution HSI (HR-HSI). In this paper, we propose a new model, named coupled tensor ring factorization (CTRF), for HSR. The proposed CTRF approach simultaneously learns high spectral resolution core tensor from the HSI and high spatial resolution core tensors from the MSI, and reconstructs the HR-HSI via tensor ring (TR) representation (Figure~\ref{fig:framework}). The CTRF model can separately exploit the low-rank property of each class (Secti…
HSRMamba: Contextual Spatial-Spectral State Space Model for Single Image Hyperspectral Super-Resolution
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University（国家多媒体软件工程研究中心，计算机学院，武汉大学）；Aerospace Information Research Institute, Henan Academy of Sciences（航天信息研究所，河南科学院）
Restoration of digital images from their degraded measurements has always been a problem of great theoretical and practical importance in numerous applications of imaging sciences. A specific solution to the problem of image restoration is generally determined by the nature of degradation phenomenon as well as by the statistical properties of measurement noises. The present study is concerned with the case in which the images of interest are corrupted by convolutional blurs and Poisson noises. To deal with such problems, there exists a range of solution methods which are based on the principle…
Due to inadequate energy captured by the hyperspectral camera sensor in poor illumination conditions, low-light hyperspectral images (HSIs) usually suffer from low visibility, spectral distortion, and various noises. A range of HSI restoration methods have been developed, yet their effectiveness in enhancing low-light HSIs is constrained. This work focuses on the low-light HSI enhancement task, which aims to reveal the spatial-spectral information hidden in darkened areas. To facilitate the development of low-light HSI processing, we collect a low-light HSI (LHSI) dataset of both indoor and ou…
Physics-Informed Untrained Learning for RGB-Guided Superresolution Single-Pixel Hyperspectral Imaging
College of Science, Northwest Agriculture & Forestry University（西北农林科技大学理学院）；Key Laboratory of Photoelectronic Imaging Technology and System, School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院光电成像技术与系统教育部重点实验室）；State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology（华中科技大学数字制造装备与技术国家重点实验室）
INRIA（法国国家信息与自动化研究所）；Département d’informatique de l’ENS（巴黎高等师范学院计算机系）；ENS（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL University（巴黎文理研究大学）
We present a novel approach to image restoration that leverages ideas from localized structured prediction and non-linear multi-task learning. We optimize a penalized energy function regularized by a sum of terms measuring the distance between patches to be restored and clean patches from an external database gathered beforehand. The resulting estimator comes with strong statistical guarantees leveraging local dependency properties of overlapping patches. We derive the corresponding algorithms for energies based on the mean-squared and Euclidean norm errors. Finally, we demonstrate the practic…
Hyperspectral images contain mixed pixels due to low spatial resolution of hyperspectral sensors. Mixed pixels are pixels containing more than one distinct material called endmembers. The presence percentages of endmembers in mixed pixels are called abundance fractions. Spectral unmixing problem refers to decomposing these pixels into a set of endmembers and abundance fractions. Due to nonnegativity constraint on abundance fractions, nonnegative matrix factorization methods (NMF) have been widely used for solving spectral unmixing problem. In this paper we have used graph regularized NMF (GNMF…
Energy-oriented Diffusion Bridge for Image Restoration with Foundational Diffusion Models
Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Department of Computer Science, City University of Hong Kong (Dongguan)（香港城市大学（东莞）计算机科学系）
Hipandas: Hyperspectral Image Joint Denoising and Super-Resolution by Image Fusion with the Panchromatic Image
Northwestern Polytechnical University（西北工业大学）；Research & Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；ETH Zürich（苏黎世联邦理工学院）；Xi’an Jiaotong University（西安交通大学）
提出零样本 Hipandas 框架，通过三网络耦合、两类低秩先验和两阶段训练，联合利用 PAN 图像完成高光谱去噪与超分辨率，性能优于现有方法。
Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Image quality is the basis of image communication and understanding tasks. Due to the blur and noise effects caused by imaging, transmission and other processes, the image quality is degraded. Blind image restoration is widely used to improve image quality, where the main goal is to faithfully estimate the blur kernel and the latent sharp image. In this study, based on experimental observation and research, an adaptively sparse regularized minimization method is originally proposed. The high-order gradients combine with low-order ones to form a hybrid regularization term, and an adaptive opera…
Nanjing Forestry University（南京林业大学）；University of Michigan（密歇根大学）；FXPAL inc.（FX帕洛阿尔托实验室公司）；Zhejiang A&F University（浙江农林大学）；University of Calgary（卡尔加里大学）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；FX Palo Alto Laboratory Inc.（FX帕洛阿尔托实验室公司）
Sliding-window based low-rank matrix approximation (LRMA) is a technique widely used in hyperspectral images (HSIs) denoising or completion. However, the uncertainty quantification of the restored HSI has not been addressed to date. Accurate uncertainty quantification of the denoised HSI facilitates to applications such as multi-source or multi-scale data fusion, data assimilation, and product uncertainty quantification, since these applications require an accurate approach to describe the statistical distributions of the input data. Therefore, we propose a prior-free closed-form element-wise …
National University of Computer and Emerging Sciences（国立计算机与新兴科学大学）；Innopolis University（英诺波利斯大学）；University of Messina（墨西拿大学）
Convolutional Neural Networks (CNN) have been rigorously studied for Hyperspectral Image Classification (HSIC) and are known to be effective in exploiting joint spatial-spectral information with the expense of lower generalization performance and learning speed due to the hard labels and non-uniform distribution over labels. Several regularization techniques have been used to overcome the aforesaid issues. However, sometimes models learn to predict the samples extremely confidently which is not good from a generalization point of view. Therefore, this paper proposed an idea to enhance the gene…
Qingdao University（青岛大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Kentucky（肯塔基大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
In this paper, we propose a novel nonconvex approach to robust principal component analysis for HSI denoising, which focuses on simultaneously developing more accurate approximations to both rank and column-wise sparsity for the low-rank and sparse components, respectively. In particular, the new method adopts the log-determinant rank approximation and a novel $\ell_{2,\log}$ norm, to restrict the local low-rank or column-wisely sparse properties for the component matrices, respectively. For the $\ell_{2,\log}$-regularized shrinkage problem, we develop an efficient, closed-form solution, which…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；International Research Center of Big Data for Sustainable Development Goals（可持续发展大数据国际研究中心）
Owing to the rapid development of sensor technology, hyperspectral (HS) remote sensing (RS) imaging has provided a significant amount of spatial and spectral information for the observation and analysis of the Earth's surface at a distance of data acquisition devices, such as aircraft, spacecraft, and satellite. The recent advancement and even revolution of the HS RS technique offer opportunities to realize the full potential of various applications, while confronting new challenges for efficiently processing and analyzing the enormous HS acquisition data. Due to the maintenance of the 3-D HS …
We propose a method for the unsupervised clustering of hyperspectral images based on spatially regularized spectral clustering with ultrametric path distances. The proposed method efficiently combines data density and geometry to distinguish between material classes in the data, without the need for training labels. The proposed method is efficient, with quasilinear scaling in the number of data points, and enjoys robust theoretical performance guarantees. Extensive experiments on synthetic and real HSI data demonstrate its strong performance compared to benchmark and state-of-the-art methods.…
Shandong University of Traditional Chinese Medicine（山东中医药大学）；Nankai University（南开大学）；ShanghaiTech University（上海科技大学）；Taishan University（泰山学院）；United Imaging Co.（联影公司）
Real-world face super-resolution (SR) is a highly ill-posed image restoration task. The fully-cycled Cycle-GAN architecture is widely employed to achieve promising performance on face SR, but prone to produce artifacts upon challenging cases in real-world scenarios, since joint participation in the same degradation branch will impact final performance due to huge domain gap between real-world and synthetic LR ones obtained by generators. To better exploit the powerful generative capability of GAN for real-world face SR, in this paper, we establish two independent degradation branches in the fo…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；The University of Hong Kong（香港大学）
Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig.1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, p…
University of Electronic Science and Technology of China（中国电子科技大学）
Recovering a large matrix from limited measurements is a challenging task arising in many real applications, such as image inpainting, compressive sensing and medical imaging, and this kind of problems are mostly formulated as low-rank matrix approximation problems. Due to the rank operator being non-convex and discontinuous, most of the recent theoretical studies use the nuclear norm as a convex relaxation and the low-rank matrix recovery problem is solved through minimization of the nuclear norm regularized problem. However, a major limitation of nuclear norm minimization is that all the sin…
Adaptive Multi-Order Graph Regularized NMF with Dual Sparsity for Hyperspectral Unmixing
School of Automation Engineering, Shanghai University of Electric Power（上海电力大学自动化工程学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机械电子工程与自动化学院）；School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能系统工程学院）
