Super-resolution reconstruction of hyperspectral images via low rank tensor modeling and total variation regularization
1 School of Mathematics；Statistics, Xi'an Jiaotong University 2 School of Mathematics；Information Science, Shaanxi Normal University 3 Shenyang Institute of Automation, Chinese Academy of Sciences
Shiying He, Haiwei Zhou, Yao Wang, Wenfei Cao, Zhi Han
In this paper, we propose a novel approach to hyperspectral image super-resolution by modeling the global spatial-and-spectral correlation and local smoothness properties over hyperspectral images. Specifically, we utilize the tensor nuclear norm and tensor folded-concave penalty functions to describe the global spatial-and-spectral correlation hidden in hyperspectral images, and 3D total variation (TV) to characterize the local spatial-and-spectral smoothness across all hyperspectral bands. Then, we develop an efficient algorithm for solving the resulting optimization problem by combing the local linear approximation (LLA) strategy and alternative direction method of multipliers (ADMM). Experimental results on one hyperspectral image dataset illustrate the merits of the proposed approach.
Fusing a low-resolution hyperspectral image (HSI) and a high-resolution multispectral image (MSI) of the same scene leads to a super-resolution image (SRI), which is information rich spatially and spectrally. In this paper, we super-resolve the HSI using the graph Laplacian defined on the MSI. Unlike many existing works, we don't assume prior knowledge about the spatial degradation from SRI to HSI, nor a perfectly aligned HSI and MSI pair. Our algorithm progressively alternates between finding the blur kernel and fusing HSI with MSI, generating accurate estimations of the blur kernel and the S…
Sparse regression methods have been proven effective in a wide range of signal processing problems such as image compression, speech coding, channel equalization, linear regression and classification. In this paper a new convex method of hyperspectral image classification is developed based on the sparse unmixing algorithm SUnSAL for which a pixel adaptive L1-norm regularization term is introduced. To further enhance class separability, the algorithm is kernelized using an RBF kernel and the final results are improved by a combination of spatial pre and post-processing operations. It is shown …
Recently, the low-rank property of different components extracted from the image has been considered in man hyperspectral image denoising methods. However, these methods usually unfold the 3D tensor to 2D matrix or 1D vector to exploit the prior information, such as nonlocal spatial self-similarity (NSS) and global spectral correlation (GSC), which break the intrinsic structure correlation of hyperspectral image (HSI) and thus lead to poor restoration quality. In addition, most of them suffer from heavy computational burden issues due to the involvement of singular value decomposition operatio…
Northwest A\&F University, College of Science, Yangling 712100, P.R. China
Several bandwise total variation (TV) regularized low-rank (LR)-based models have been proposed to remove mixed noise in hyperspectral images (HSIs). Conventionally, the rank of LR matrix is approximated using nuclear norm (NN). The NN is defined by adding all singular values together, which is essentially a $L_1$-norm of the singular values. It results in non-negligible approximation errors and thus the resulting matrix estimator can be significantly biased. Moreover, these bandwise TV-based methods exploit the spatial information in a separate manner. To cope with these problems, we propose …
The spatio-spectral total variation (SSTV) model has been widely used as an effective regularization of hyperspectral images (HSI) for various applications such as mixed noise removal. However, since SSTV computes local spatial differences uniformly, it is difficult to remove noise while preserving complex spatial structures with fine edges and textures, especially in situations of high noise intensity. To solve this problem, we propose a new TV-type regularization called Graph-SSTV (GSSTV), which generates a graph explicitly reflecting the spatial structure of the target HSI from noisy HSIs a…
Chinese University of Hongkong（香港中文大学）；University of Electronic Science and Technology of China（电子科技大学）
Hyperspectral images (HSIs) are often corrupted by a mixture of several types of noise during the acquisition process, e.g., Gaussian noise, impulse noise, dead lines, stripes, and many others. Such complex noise could degrade the quality of the acquired HSIs, limiting the precision of the subsequent processing. In this paper, we present a novel tensor-based HSI restoration approach by fully identifying the intrinsic structures of the clean HSI part and the mixed noise part respectively. Specifically, for the clean HSI part, we use tensor Tucker decomposition to describe the global correlation…
Qingdao University（青岛大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Kentucky（肯塔基大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
In this paper, we propose a novel nonconvex approach to robust principal component analysis for HSI denoising, which focuses on simultaneously developing more accurate approximations to both rank and column-wise sparsity for the low-rank and sparse components, respectively. In particular, the new method adopts the log-determinant rank approximation and a novel $\ell_{2,\log}$ norm, to restrict the local low-rank or column-wisely sparse properties for the component matrices, respectively. For the $\ell_{2,\log}$-regularized shrinkage problem, we develop an efficient, closed-form solution, which…
Preventing Overfitting in Deep Image Prior for Hyperspectral Image Denoising
Robotics Institute, Athena Research and Innovation Center（雅典娜研究与创新中心机器人研究所）；School of Electrical and Computer Engineering, National Technical University of Athens（雅典国家技术大学电气与计算机工程学院）
University of Electronic Science and Technology of China（电子科技大学）
Hyperspectral image has become increasingly crucial due to its abundant spectral information. However, It has poor spatial resolution with the limitation of the current imaging mechanism. Nowadays, many convolutional neural networks have been proposed for the hyperspectral image super-resolution problem. However, convolutional neural network (CNN) based methods only consider the local information instead of the global one with the limited kernel size of receptive field in the convolution operation. In this paper, we design a network based on the transformer for fusing the low-resolution hypers…
The Norwegian Colour and Visual Computing Laboratory（挪威色彩与视觉计算实验室）；NTNU（挪威科技大学）
Deep learning algorithms have demonstrated state-of-the-art performance in various tasks of image restoration. This was made possible through the ability of CNNs to learn from large exemplar sets. However, the latter becomes an issue for hyperspectral image processing where datasets commonly consist of just a few images. In this work, we propose a new approach to denoising, inpainting, and super-resolution of hyperspectral image data using intrinsic properties of a CNN without any training. The performance of the given algorithm is shown to be comparable to the performance of trained networks,…
Spectral imaging enables spatially-resolved identification of materials in remote sensing, biomedicine, and astronomy. However, acquisition times require balancing spectral and spatial resolution with signal-to-noise. Hyperspectral imaging provides superior material specificity, while multispectral images are faster to collect at greater fidelity. We propose an approach for fusing hyperspectral and multispectral images to provide high-quality hyperspectral output. The proposed optimization leverages the least absolute shrinkage and selection operator (LASSO) to perform variable selection and r…
Hyperspectral imaging is a cutting-edge type of remote sensing used for mapping vegetation properties, rock minerals and other materials. A major drawback of hyperspectral imaging devices is their intrinsic low spatial resolution. In this paper, we propose a method for increasing the spatial resolution of a hyperspectral image by fusing it with an image of higher spatial resolution that was obtained with a different imaging modality. This is accomplished by solving a variational problem in which the regularization functional is the directional total variation. To accommodate for possible mis-r…
A TRPCA-Inspired Deep Unfolding Network for Hyperspectral Image Denoising via Thresholded t-SVD and Top-K Sparse Transformer
College of Computer Science and Engineering, Shandong University of Science and Technology（计算机科学与工程学院，山东科技大学）；Gosci Information Technology Co.,Ltd.（Gosci信息技术有限公司）
In this study, the problem of computing a sparse representation of multi-dimensional visual data is considered. In general, such data e.g., hyperspectral images, color images or video data consists of signals that exhibit strong local dependencies. A new computationally efficient sparse coding optimization problem is derived by employing regularization terms that are adapted to the properties of the signals of interest. Exploiting the merits of the learnable regularization techniques, a neural network is employed to act as structure prior and reveal the underlying signal dependencies. To solve…
Harbin Institute of Technology（哈尔滨工业大学）；Wuhan University（武汉大学）
High-resolution (HR) hyperspectral face image plays an important role in face related computer vision tasks under uncontrolled conditions, such as low-light environment and spoofing attacks. However, the dense spectral bands of hyperspectral face images come at the cost of limited amount of photons reached a narrow spectral window on average, which greatly reduces the spatial resolution of hyperspectral face images. In this paper, we investigate how to adapt the deep learning techniques to hyperspectral face image super-resolution (HFSR), especially when the training samples are very limited. …
There is much current interest in using multi-sensor airborne remote sensing to monitor the structure and biodiversity of forests. This paper addresses the application of non-parametric image registration techniques to precisely align images obtained from multimodal imaging, which is critical for the successful identification of individual trees using object recognition approaches. Non-parametric image registration, in particular the technique of optimizing one objective function containing data fidelity and regularization terms, provides flexible algorithms for image registration. Using a sur…
Different from traditional hyperspectral super-resolution approaches that focus on improving the spatial resolution, spectral super-resolution aims at producing a high-resolution hyperspectral image from the RGB observation with super-resolution in spectral domain. However, it is challenging to accurately reconstruct a high-dimensional continuous spectrum from three discrete intensity values at each pixel, since too much information is lost during the procedure where the latent hyperspectral image is downsampled (e.g., with x10 scaling factor) in spectral domain to produce an RGB observation. …
This paper introduces a graph Laplacian regularization in the hyperspectral unmixing formulation. The proposed regularization relies upon the construction of a graph representation of the hyperspectral image. Each node in the graph represents a pixel's spectrum, and edges connect spectrally and spatially similar pixels. The proposed graph framework promotes smoothness in the estimated abundance maps and collaborative estimation between homogeneous areas of the image. The resulting convex optimization problem is solved using the Alternating Direction Method of Multipliers (ADMM). A special atte…
Deep learning based methods have achieved remarkable success in image restoration and enhancement, but most such methods rely on RGB input images. These methods fail to take into account the rich spectral distribution of natural images. We propose a deep architecture, SpecNet, which computes spectral profile to estimate pixel-wise dynamic range adjustment of a given image. First, we employ an unpaired cycle-consistent framework to generate hyperspectral images (HSI) from low-light input images. HSI is further used to generate a normal light image of the same scene. We incorporate a self-superv…
The paper proposes a new high spatial resolution hyperspectral (HR-HS) image estimation method based on convex optimization. The method assumes a low spatial resolution HS (LR-HS) image and a guide image as observations, where both observations are contaminated by noise. Our method simultaneously estimates an HR-HS image and a noiseless guide image, so the method can utilize spatial information in a guide image even if it is contaminated by heavy noise. The proposed estimation problem adopts hybrid spatio-spectral total variation as regularization and evaluates the edge similarity between HR-H…
Federal University of Santa Catarina（圣卡塔琳娜联邦大学）；Northeastern University（东北大学）；Catholic University of Pelotas（佩洛塔斯天主教大学）
Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral-multispectral (HS-MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images, circumventing the main limitation of this imaging modality. Existing HS-MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent mate…
Hyperspectral super-resolution (HSR) fuses a low-resolution hyperspectral image (HSI) and a high-resolution multispectral image (MSI) to obtain a high-resolution HSI (HR-HSI). In this paper, we propose a new model, named coupled tensor ring factorization (CTRF), for HSR. The proposed CTRF approach simultaneously learns high spectral resolution core tensor from the HSI and high spatial resolution core tensors from the MSI, and reconstructs the HR-HSI via tensor ring (TR) representation (Figure~\ref{fig:framework}). The CTRF model can separately exploit the low-rank property of each class (Secti…
The University of Tennessee（田纳西大学）；Applied Research LLC（应用研究有限责任公司）
In many computer vision applications, obtaining images of high resolution in both the spatial and spectral domains are equally important. However, due to hardware limitations, one can only expect to acquire images of high resolution in either the spatial or spectral domains. This paper focuses on hyperspectral image super-resolution (HSI-SR), where a hyperspectral image (HSI) with low spatial resolution (LR) but high spectral resolution is fused with a multispectral image (MSI) with high spatial resolution (HR) but low spectral resolution to obtain HR HSI. Existing deep learning-based solution…
Pan-denoising: Guided Hyperspectral Image Denoising via Weighted Represent Coefficient Total Variation
School of Mathematics and Statistics, Northwestern Polytechnical University（西北工业大学数学与统计学院）；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
Tensor-based methods have recently emerged as a more natural and effective formulation to address many problems in hyperspectral imaging. In hyperspectral unmixing (HU), low-rank constraints on the abundance maps have been shown to act as a regularization which adequately accounts for the multidimensional structure of the underlying signal. However, imposing a strict low-rank constraint for the abundance maps does not seem to be adequate, as important information that may be required to represent fine scale abundance behavior may be discarded. This paper introduces a new low-rank tensor regula…
Commonwealth Scientific and Industrial Research Organisation（澳大利亚联邦科学与工业研究组织）
Natural images tend to mostly consist of smooth regions with individual pixels having highly correlated spectra. This information can be exploited to recover hyperspectral images of natural scenes from their incomplete and noisy measurements. To perform the recovery while taking full advantage of the prior knowledge, we formulate a composite cost function containing a square-error data-fitting term and two distinct regularization terms pertaining to spatial and spectral domains. The regularization for the spatial domain is the sum of total-variation of the image frames corresponding to all spe…
Hyperspectral image (HSI) denoising aims to restore clean HSI from the noise-contaminated one. Noise contamination can often be caused during data acquisition and conversion. In this paper, we propose a novel spatial-spectral total variation (SSTV) regularized nonconvex local low-rank (LR) tensor approximation method to remove mixed noise in HSIs. From one aspect, the clean HSI data have its underlying local LR tensor property, even though the real HSI data may not be globally low-rank due to out-liers and non-Gaussian noise. According to this fact, we propose a novel tensor $L_γ$-norm to form…
Supervised learning with a convolutional neural network is recognized as a powerful means of image restoration. However, most such methods have been designed for application to grayscale and/or color images; therefore, they have limited success when applied to hyperspectral image restoration. This is partially owing to large datasets being difficult to collect, and also the heavy computational load associated with the restoration of an image with many spectral bands. To address this difficulty, we propose a novel self-supervised learning strategy for application to hyperspectral image restorat…
We present a supervised hyperspectral image segmentation algorithm based on a convex formulation of a marginal maximum a posteriori segmentation with hidden fields and structure tensor regularization: Segmentation via the Constraint Split Augmented Lagrangian Shrinkage by Structure Tensor Regularization (SegSALSA-STR). This formulation avoids the generally discrete nature of segmentation problems and the inherent NP-hardness of the integer optimization associated. We extend the Segmentation via the Constraint Split Augmented Lagrangian Shrinkage (SegSALSA) algorithm by generalizing the vectori…
With the development of deep learning technology, multi-spectral image super-resolution methods based on convolutional neural network have recently achieved great progress. However, the single hyperspectral image super-resolution remains a challenging problem due to the high-dimensional and complex spectral characteristics of hyperspectral data, which make it difficult to simultaneously capture spatial and spectral information. To deal with this issue, we propose a novel Feedback Refined Local-Global Network (FRLGN) for the super-resolution of hyperspectral image. To be specific, we develop a …
Physics-Informed Untrained Learning for RGB-Guided Superresolution Single-Pixel Hyperspectral Imaging
College of Science, Northwest Agriculture & Forestry University（西北农林科技大学理学院）；Key Laboratory of Photoelectronic Imaging Technology and System, School of Optics and Photonics, Beijing Institute of Technology（北京理工大学光电学院光电成像技术与系统教育部重点实验室）；State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology（华中科技大学数字制造装备与技术国家重点实验室）
Deep learning-based hyperspectral image super-resolution (SR) methods have achieved great success recently. However, most existing models can not effectively explore spatial information and spectral information between bands simultaneously, obtaining relatively low performance. To address this issue, in this paper, we propose a novel spectral-spatial residual network for hyperspectral image super-resolution (SSRNet). Our method can effectively explore spatial-spectral information by using 3D convolution instead of 2D convolution, which enables the network to better extract potential informatio…
Northwestern Polytechnical University（西北工业大学）；University of New South Wales（新南威尔士大学）
Hyperspectral image produces high spectral resolution at the sacrifice of spatial resolution. Without reducing the spectral resolution, improving the resolution in the spatial domain is a very challenging problem. Motivated by the discovery that hyperspectral image exhibits high similarity between adjacent bands in a large spectral range, in this paper, we explore a new structure for hyperspectral image super-resolution (DualSR), leading to a dual-stage design, i.e., coarse stage and fine stage. In coarse stage, five bands with high similarity in a certain spectral range are divided into three…
Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspec…
Harbin Institute of Technology（哈尔滨工业大学）；Wuhan University（武汉大学）
Recently, single gray/RGB image super-resolution reconstruction task has been extensively studied and made significant progress by leveraging the advanced machine learning techniques based on deep convolutional neural networks (DCNNs). However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-dimensional and complex spectral patterns in hyperspectral image. In this paper, we make a step forward by investigating how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches for computati…
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan University（武汉大学）
To solve the ill-posed problem of hyperspectral image super-resolution (HSISR), an usually method is to use the prior information of the hyperspectral images (HSIs) as a regularization term to constrain the objective function. Model-based methods using hand-crafted priors cannot fully characterize the properties of HSIs. Learning-based methods usually use a convolutional neural network (CNN) to learn the implicit priors of HSIs. However, the learning ability of CNN is limited, it only considers the spatial characteristics of the HSIs and ignores the spectral characteristics, and convolution is…
Towards Lightweight Hyperspectral Image Super-Resolution with Depthwise Separable Dilated Convolutional Network
Department of Computer Science, Aalto University, Finland（艾尔沃斯大学计算机科学系）；School of Electrical and Electronic Engineering, University of Sheffield, United Kingdom（谢菲尔德大学电子与电气工程学院）
Federal University of Santa Catarina（圣卡塔琳娜联邦大学）；CNRS（法国国家科学研究中心）；OCA（蔚蓝海岸天文台）
Sparse hyperspectral unmixing from large spectral libraries has been considered to circumvent limitations of endmember extraction algorithms in many applications. This strategy often leads to ill-posed inverse problems, which can benefit from spatial regularization strategies. While existing spatial regularization methods improve the problem conditioning and promote piecewise smooth solutions, they lead to large nonsmooth optimization problems. Thus, efficiently introducing spatial context in the unmixing problem remains a challenge, and a necessity for many real world applications. In this pa…
To overcome inherent hardware limitations of hyperspectral imaging systems with respect to their spatial resolution, fusion-based hyperspectral image (HSI) super-resolution is attracting increasing attention. This technique aims to fuse a low-resolution (LR) HSI and a conventional high-resolution (HR) RGB image in order to obtain an HR HSI. Recently, deep learning architectures have been used to address the HSI super-resolution problem and have achieved remarkable performance. However, they ignore the degradation model even though this model has a clear physical interpretation and may contribu…
University of Electronic Science and Technology of China（电子科技大学）；Southwestern University of Finance and Economics（西南财经大学）；University of Salerno（萨勒诺大学）；Institute of Methodologies for Environmental Analysis, CNR-IMAA（意大利国家研究委员会环境分析方法研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）
Hyperspectral images are of crucial importance in order to better understand features of different materials. To reach this goal, they leverage on a high number of spectral bands. However, this interesting characteristic is often paid by a reduced spatial resolution compared with traditional multispectral image systems. In order to alleviate this issue, in this work, we propose a simple and efficient architecture for deep convolutional neural networks to fuse a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution hyperspectr…
Efficient Feedback Gate Network for Hyperspectral Image Super-Resolution
School of Electronic and Information Engineering, Anhui University（安徽大学电子与信息工程学院）；School of Artificial intelligence, Anhui University（安徽大学人工智能学院）；Institutes of Physical Science, Chinese Academy Sciences（中国科学院物理科学研究院）
