The recently developed transformer networks have achieved impressive performance in image denoising by exploiting the self-attention (SA) in images. However, the existing methods mostly use a relatively small window to compute SA due to the quadratic complexity of it, which limits the model's ability to model long-term image information. In this paper, we propose the spatial-frequency attention network (SFANet) to enhance the network's ability in exploiting long-range dependency. For spatial attention module (SAM), we adopt dilated SA to model long-range dependency. In the frequency attention module (FAM), we exploit more global information by using Fast Fourier Transform (FFT) by designing a window-based frequency channel attention (WFCA) block to effectively model deep frequency features and their dependencies. To make our module applicable to images of different sizes and keep the model consistency between training and inference, we apply window-based FFT with a set of fixed window sizes. In addition, channel attention is computed on both real and imaginary parts of the Fourier spectrum, which further improves restoration performance. The proposed WFCA block can effectively model image long-range dependency with acceptable complexity. Experiments on multiple denoising benchmarks demonstrate the leading performance of SFANet network.
Video Denoising is one of the fundamental tasks of any videoprocessing pipeline. It is different from image denoising due to the tem-poral aspects of video frames, and any image denoising approach appliedto videos will result in flickering. The proposed method makes use oftemporal as well as spatial dimensions of video frames as part of a two-stage pipeline. Each stage in the architecture named as Spatio-TemporalNetwork uses a channel-wise attention mechanism to forward the encodersignal to the decoder side. The Attention Block used in this paper usessoft attention to ranks the filters for bet…
Northern Arizona University（北亚利桑那大学）；Xi’an Network Computing Data Technology Co., Ltd.（西安网络计算数据技术有限公司）
Convolutional layers in Artificial Neural Networks (ANN) treat the channel features equally without feature selection flexibility. While using ANNs for image denoising in real-world applications with unknown noise distributions, particularly structured noise with learnable patterns, modeling informative features can substantially boost the performance. Channel attention methods in real image denoising tasks exploit dependencies between the feature channels, hence being a frequency component filtering mechanism. Existing channel attention modules typically use global statics as descriptors to l…
Wuhan University（武汉大学）；University of Arkansas（阿肯色大学）；Wuhan Institute of Technology（武汉工程大学）
Hyperspectral images (HSIs) have been widely used in a variety of applications thanks to the rich spectral information they are able to provide. Among all HSI processing tasks, HSI denoising is a crucial step. Recently, deep learning-based image denoising methods have made great progress and achieved great performance. However, existing methods tend to ignore the correlations between adjacent spectral bands, leading to problems such as spectral distortion and blurred edges in denoised results. In this study, we propose a novel HSI denoising network, termed SSCAN, that combines group convolutio…
Southeast University（东南大学）；Jiangsu Industrial Technology Research Institute（江苏集萃工业技术研究院）
Deep learning based image denoising methods have been extensively investigated. In this paper, attention mechanism enhanced kernel prediction networks (AME-KPNs) are proposed for burst image denoising, in which, nearly cost-free attention modules are adopted to first refine the feature maps and to further make a full use of the inter-frame and intra-frame redundancies within the whole image burst. The proposed AME-KPNs output per-pixel spatially-adaptive kernels, residual maps and corresponding weight maps, in which, the predicted kernels roughly restore clean pixels at their corresponding loc…
Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；The Australian National University（澳大利亚国立大学）
Deep convolutional neural networks perform better on images containing spatially invariant noise (synthetic noise); however, their performance is limited on real-noisy photographs and requires multiple stage network modeling. To advance the practicability of denoising algorithms, this paper proposes a novel single-stage blind real image denoising network (RIDNet) by employing a modular architecture. We use a residual on the residual structure to ease the flow of low-frequency information and apply feature attention to exploit the channel dependencies. Furthermore, the evaluation in terms of qu…
Hyperspectral image (HSI) denoising is a crucial preprocessing procedure to improve the performance of the subsequent HSI interpretation and applications. In this paper, a novel deep learning-based method for this task is proposed, by learning a non-linear end-to-end mapping between the noisy and clean HSIs with a combined spatial-spectral deep convolutional neural network (HSID-CNN). Both the spatial and spectral information are simultaneously assigned to the proposed network. In addition, multi-scale feature extraction and multi-level feature representation are respectively employed to captu…
Hyperspectral image (HSI) denoising is a crucial preprocessing procedure for the subsequent HSI applications. Unfortunately, though witnessing the development of deep learning in HSI denoising area, existing convolution-based methods face the trade-off between computational efficiency and capability to model non-local characteristics of HSI. In this paper, we propose a Spatial-Spectral Transformer (SST) to alleviate this problem. To fully explore intrinsic similarity characteristics in both spatial dimension and spectral dimension, we conduct non-local spatial self-attention and global spectra…
Hyperspectral image (HSI) denoising is of crucial importance for many subsequent applications, such as HSI classification and interpretation. In this paper, we propose an attention-based deep residual network to directly learn a mapping from noisy HSI to the clean one. To jointly utilize the spatial-spectral information, the current band and its $K$ adjacent bands are simultaneously exploited as the input. Then, we adopt convolution layer with different filter sizes to fuse the multi-scale feature, and use shortcut connection to incorporate the multi-level information for better noise removal.…
Jilin University（吉林大学）；University of Trento（特伦托大学）；ETH Zurich（苏黎世联邦理工学院）
Degraded images commonly exist in the general sources of character images, leading to unsatisfactory character recognition results. Existing methods have dedicated efforts to restoring degraded character images. However, the denoising results obtained by these methods do not appear to improve character recognition performance. This is mainly because current methods only focus on pixel-level information and ignore critical features of a character, such as its glyph, resulting in character-glyph damage during the denoising process. In this paper, we introduce a novel generic framework based on g…
Hybrid Convolutional and Attention Network for Hyperspectral Image Denoising
School of Computer Science and Technology, Ocean University of China（中国海洋大学计算机科学与技术学院）；Department of Electrical and Computer Engineering, Mississippi State University（密西西比州立大学电气与计算机工程系）
Recently, convolutional neural networks (CNNs) and attention mechanisms have been widely used in image denoising and achieved satisfactory performance. However, the previous works mostly use a single head to receive the noisy image, limiting the richness of extracted features. Therefore, a novel CNN with multiple heads (MH) named MHCNN is proposed in this paper, whose heads will receive the input images rotated by different rotation angles. MH makes MHCNN simultaneously utilize features of rotated images to remove noise. To integrate these features effectively, we present a novel multi-path at…
Harbin Institute of Technology（哈尔滨工业大学）；Guangdong University of Technology（广东工业大学）
Since the proposal of big data analysis and Graphic Processing Unit (GPU), the deep learning technology has received a great deal of attention and has been widely applied in the field of imaging processing. In this paper, we have an aim to completely review and summarize the deep learning technologies for image denoising proposed in recent years. Morever, we systematically analyze the conventional machine learning methods for image denoising. Finally, we point out some research directions for the deep learning technologies in image denoising.
Previous works have shown that convolutional neural networks can achieve good performance in image denoising tasks. However, limited by the local rigid convolutional operation, these methods lead to oversmoothing artifacts. A deeper network structure could alleviate these problems, but more computational overhead is needed. In this paper, we propose a novel spatial-adaptive denoising network (SADNet) for efficient single image blind noise removal. To adapt to changes in spatial textures and edges, we design a residual spatial-adaptive block. Deformable convolution is introduced to sample the s…
Rochester Institute of Technology（罗切斯特理工学院）；Chosun University（朝鲜大学）
Unpaired image denoising has achieved promising development over the last few years. Regardless of the performance, methods tend to heavily rely on underlying noise properties or any assumption which is not always practical. Alternatively, if we can ground the problem from a structural perspective rather than noise statistics, we can achieve a more robust solution. with such motivation, we propose a self-supervised denoising scheme that is unpaired and relies on spatial degradation followed by a regularized refinement. Our method shows considerable improvement over previous methods and exhibit…
Image denoising is always a challenging task in the field of computer vision and image processing. In this paper, we have proposed an encoder-decoder model with direct attention, which is capable of denoising and reconstruct highly corrupted images. Our model consists of an encoder and a decoder, where the encoder is a convolutional neural network and decoder is a multilayer Long Short-Term memory network. In the proposed model, the encoder reads an image and catches the abstraction of that image in a vector, where decoder takes that vector as well as the corrupted image to reconstruct a clean…
In this paper, we propose a novel image denoising algorithm exploiting features from both spatial as well as transformed domain. We implement intensity-invariance based improved grouping for collaborative support-agnostic sparse reconstruction. For collaboration firstly, we stack similar-structured patches via intensity-invariant correlation measure. The grouped patches collaborate to yield desirable sparse estimates for noise filtering. This is because similar patches share the same support in the transformed domain, such similar supports can be used as probabilities of active taps to refine …
Nanjing University of Science and Technology（南京理工大学）；University of Macau（澳门大学）；Griffith University（格里菲斯大学）；Zhejiang University（浙江大学）
Deep learning (DL) based hyperspectral images (HSIs) denoising approaches directly learn the nonlinear mapping between observed noisy images and underlying clean images. They normally do not consider the physical characteristics of HSIs, therefore making them lack of interpretability that is key to understand their denoising mechanism.. In order to tackle this problem, we introduce a novel model guided interpretable network for HSI denoising. Specifically, fully considering the spatial redundancy, spectral low-rankness and spectral-spatial properties of HSIs, we first establish a subspace base…
Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）
Due to the fast inference and good performance, discriminative learning methods have been widely studied in image denoising. However, these methods mostly learn a specific model for each noise level, and require multiple models for denoising images with different noise levels. They also lack flexibility to deal with spatially variant noise, limiting their applications in practical denoising. To address these issues, we present a fast and flexible denoising convolutional neural network, namely FFDNet, with a tunable noise level map as the input. The proposed FFDNet works on downsampled sub-imag…
Carnegie Mellon University（卡内基梅隆大学）；University of British Columbia（不列颠哥伦比亚大学）；SenseTime Research（商汤科技研究院）；University of California, Merced（加州大学默塞德分校）
Existing denoising methods typically restore clear results by aggregating pixels from the noisy input. Instead of relying on hand-crafted aggregation schemes, we propose to explicitly learn this process with deep neural networks. We present a spatial pixel aggregation network and learn the pixel sampling and averaging strategies for image denoising. The proposed model naturally adapts to image structures and can effectively improve the denoised results. Furthermore, we develop a spatio-temporal pixel aggregation network for video denoising to efficiently sample pixels across the spatio-tempora…
Northeastern University at Qinhuangdao（东北大学秦皇岛分校）；Univ Bretagne-Sud（南布列塔尼大学）；Northeastern University（东北大学）；Tianjin University（天津大学）
Following the pioneering works of Rudin, Osher and Fatemi on total variation (TV) and of Buades, Coll and Morel on non-local means (NL-means), the last decade has seen a large number of denoising methods mixing these two approaches, starting with the nonlocal total variation (NLTV) model. The present article proposes an analysis of the NLTV model for image denoising as well as a number of improvements, the most important of which being to apply the denoising both in the space domain and in the Fourier domain, in order to exploit the complementarity of the representation of image data in both d…
Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）；Graz University of Technology（格拉茨技术大学）；Xi’an Jiaotong University（西安交通大学）
Discriminative model learning for image denoising has been recently attracting considerable attentions due to its favorable denoising performance. In this paper, we take one step forward by investigating the construction of feed-forward denoising convolutional neural networks (DnCNNs) to embrace the progress in very deep architecture, learning algorithm, and regularization method into image denoising. Specifically, residual learning and batch normalization are utilized to speed up the training process as well as boost the denoising performance. Different from the existing discriminative denois…
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）
Significant progress has been made in self-supervised image denoising (SSID) in the recent few years. However, most methods focus on dealing with spatially independent noise, and they have little practicality on real-world sRGB images with spatially correlated noise. Although pixel-shuffle downsampling has been suggested for breaking the noise correlation, it breaks the original information of images, which limits the denoising performance. In this paper, we propose a novel perspective to solve this problem, i.e., seeking for spatially adaptive supervision for real-world sRGB image denoising. …
Hyperspectral image denoising is unique for the highly similar and correlated spectral information that should be properly considered. However, existing methods show limitations in exploring the spectral correlations across different bands and feature interactions within each band. Besides, the low- and high-level features usually exhibit different importance for different spatial-spectral regions, which is not fully explored for current algorithms as well. In this paper, we present a Mixed Attention Network (MAN) that simultaneously considers the inter- and intra-spectral correlations as well…
SSUMamba: Spatial-Spectral Selective State Space Model for Hyperspectral Image Denoising
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）
该研究提出 SSUMamba，通过六种空间-光谱连续扫描顺序、双向 SSM 与 3D 卷积高效建模高光谱图像的长程及局部依赖，在降低每批内存消耗的同时提升去噪性能。
Hyperspectral image (HSI) denoising aims to restore clean HSI from the noise-contaminated one. Noise contamination can often be caused during data acquisition and conversion. In this paper, we propose a novel spatial-spectral total variation (SSTV) regularized nonconvex local low-rank (LR) tensor approximation method to remove mixed noise in HSIs. From one aspect, the clean HSI data have its underlying local LR tensor property, even though the real HSI data may not be globally low-rank due to out-liers and non-Gaussian noise. According to this fact, we propose a novel tensor $L_γ$-norm to form…
Hamamatsu Photonics K. K.（滨松光子学株式会社）；Hamamatsu University School of Medicine（滨松医科大学）
Although supervised convolutional neural networks (CNNs) often outperform conventional alternatives for denoising positron emission tomography (PET) images, they require many low- and high-quality reference PET image pairs. Herein, we propose an unsupervised 3D PET image denoising method based on an anatomical information-guided attention mechanism. The proposed magnetic resonance-guided deep decoder (MR-GDD) utilizes the spatial details and semantic features of MR-guidance image more effectively by introducing encoder-decoder and deep decoder subnetworks. Moreover, the specific shapes and pat…
Qingdao University（青岛大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Kentucky（肯塔基大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
In this paper, we propose a novel nonconvex approach to robust principal component analysis for HSI denoising, which focuses on simultaneously developing more accurate approximations to both rank and column-wise sparsity for the low-rank and sparse components, respectively. In particular, the new method adopts the log-determinant rank approximation and a novel $\ell_{2,\log}$ norm, to restrict the local low-rank or column-wisely sparse properties for the component matrices, respectively. For the $\ell_{2,\log}$-regularized shrinkage problem, we develop an efficient, closed-form solution, which…
Learning Multi-scale Spatial-frequency Features for Image Denoising
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering（计算机科学与工程学院）；School of Intelligence Science and Technology（智能科学与技术学院）
The paper proposes a new high spatial resolution hyperspectral (HR-HS) image estimation method based on convex optimization. The method assumes a low spatial resolution HS (LR-HS) image and a guide image as observations, where both observations are contaminated by noise. Our method simultaneously estimates an HR-HS image and a noiseless guide image, so the method can utilize spatial information in a guide image even if it is contaminated by heavy noise. The proposed estimation problem adopts hybrid spatio-spectral total variation as regularization and evaluates the edge similarity between HR-H…
Since convolutional neural networks perform well in learning generalizable image priors from large-scale data, these models have been widely used in image denoising tasks. However, the computational complexity increases dramatically as well on complex model. In this paper, We propose a novel lightweight Complementary Attention Module, which includes a density module and a sparse module, which can cooperatively mine dense and sparse features for feature complementary learning to build an efficient lightweight architecture. Moreover, to reduce the loss of details caused by denoising, this paper …
Imaging under photon-scarce situations introduces challenges to many applications as the captured images are with low signal-to-noise ratio and poor luminance. In this paper, we investigate the raw image restoration under low-photon-count conditions by simulating the imaging of quanta image sensor (QIS). We develop a lightweight framework, which consists of a multi-level pyramid denoising network (MPDNet) and a luminance adjustment (LA) module to achieve separate denoising and luminance enhancement. The main component of our framework is the multi-skip attention residual block (MARB), which in…
Convolutional neural network (CNN)-based image denoising methods have been widely studied recently, because of their high-speed processing capability and good visual quality. However, most of the existing CNN-based denoisers learn the image prior from the spatial domain, and suffer from the problem of spatially variant noise, which limits their performance in real-world image denoising tasks. In this paper, we propose a discrete wavelet denoising CNN (WDnCNN), which restores images corrupted by various noise with a single model. Since most of the content or energy of natural images resides in …
School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学学院）；Department of Computer and Information Science, Faculty of Science and Technology, University of Macau（澳门大学计算机与信息科学系）；Hubei Center for Applied Mathematics, Wuhan University（湖北应用数学中心，武汉大学）；Hubei Key Laboratory of Computational Science, Wuhan University（湖北计算科学重点实验室，武汉大学）
Deep Equilibrium Convolutional Sparse Coding for Hyperspectral Image Denoising
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；College of Computer Science, Zhejiang University（浙江大学计算机科学学院）
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
The development of neural networks has greatly improved the performance in various computer vision tasks. In the filed of image denoising, convolutional neural network based methods such as DnCNN break through the limits of classical methods, achieving better quantitative results. However, the epistemic uncertainty existing in neural networks limits further improvements in their performance over denoising tasks. Therefore, we develop and study different solutions to minimize uncertainty and further improve the removal of noise. From the perspective of ensemble learning, we implement manipulati…
School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）；The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）；Shanxi Engineering Research Center of Computational Imaging and Medical Intelligence（山西省计算成像与医学智能工程研究中心）
Data61-Commonwealth Scientific Industrial Research Organization (CSIRO)（Data61-联邦科学与工业研究组织）；Australian National University（澳大利亚国立大学）
Deep convolutional neural networks perform better on images containing spatially invariant degradations, also known as synthetic degradations; however, their performance is limited on real-degraded photographs and requires multiple-stage network modeling. To advance the practicability of restoration algorithms, this paper proposes a novel single-stage blind real image restoration network (R$^2$Net) by employing a modular architecture. We use a residual on the residual structure to ease the flow of low-frequency information and apply feature attention to exploit the channel dependencies. Furthe…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）
Most image denoising networks apply a single set of static convolutional kernels across the entire input image. This is sub-optimal for natural images, as they often consist of heterogeneous visual patterns. Dynamic convolution tries to address this issue by using per-pixel convolution kernels, but this greatly increases computational cost. In this work, we present Malleable Convolution (MalleConv), which performs spatial-varying processing with minimal computational overhead. MalleConv uses a smaller set of spatially-varying convolution kernels, a compromise between static and per-pixel convo…
