From Group Sparse Coding to Rank Minimization: A Novel Denoising Model for Low-level Image Restoration
College of Electronic and Optical Engineering & College of Microelectronics, Nanjing University of Posts and Telecommunications（南京邮电大学电子与光学工程学院、微电子学院）；College of Telecommunications & Information Engineering, Nanjing University of Posts and Telecommunications（南京邮电大学通信与信息工程学院）
Recently, low-rank matrix recovery theory has been emerging as a significant progress for various image processing problems. Meanwhile, the group sparse coding (GSC) theory has led to great successes in image restoration (IR) problem with each group contains low-rank property. In this paper, we propose a novel low-rank minimization based denoising model for IR tasks under the perspective of GSC, an important connection between our denoising model and rank minimization problem has been put forward. To overcome the bias problem caused by convex nuclear norm minimization (NNM) for rank approximation, a more generalized and flexible rank relaxation function is employed, namely weighted nonconvex relaxation. Accordingly, an efficient iteratively-reweighted algorithm is proposed to handle the resulting minimization problem combing with the popular L_(1/2) and L_(2/3) thresholding operators. Finally, our proposed denoising model is applied to IR problems via an alternating direction method of multipliers (ADMM) strategy. Typical IR experiments on image compressive sensing (CS), inpainting, deblurring and impulsive noise removal demonstrate that our proposed method can achieve significantly higher PSNR/FSIM values than many relevant state-of-the-art methods.
Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sa…
Diffusion models can be used as learned priors for solving various inverse problems. However, most existing approaches are restricted to linear inverse problems, limiting their applicability to more general cases. In this paper, we build upon Denoising Diffusion Restoration Models (DDRM) and propose a method for solving some non-linear inverse problems. We leverage the pseudo-inverse operator used in DDRM and generalize this concept for other measurement operators, which allows us to use pre-trained unconditional diffusion models for applications such as JPEG artifact correction. We empiricall…
This paper presents a novel method for restoring digital videos via a Deep Plug-and-Play (PnP) approach. Under a Bayesian formalism, the method consists in using a deep convolutional denoising network in place of the proximal operator of the prior in an alternating optimization scheme. We distinguish ourselves from prior PnP work by directly applying that method to restore a digital video from a degraded video observation. This way, a network trained once for denoising can be repurposed for other video restoration tasks. Our experiments in video deblurring, super-resolution, and interpolation …
Wuhan University（武汉大学）；JD Explore Academy（京东探索研究院）；University of Sydney（悉尼大学）
Diffusion models have recently received a surge of interest due to their impressive performance for image restoration, especially in terms of noise robustness. However, existing diffusion-based methods are trained on a large amount of training data and perform very well in-distribution, but can be quite susceptible to distribution shift. This is especially inappropriate for data-starved hyperspectral image (HSI) restoration. To tackle this problem, this work puts forth a self-supervised diffusion model for HSI restoration, namely Denoising Diffusion Spatio-Spectral Model (\texttt{DDS2M}), whic…
National Chung Hsing University（国立中兴大学）；National Taichung University of Science and Technology（国立台中科技大学）
Image restoration is a challenging ill-posed problem which also has been a long-standing issue. In the past few years, the convolution neural networks (CNNs) almost dominated the computer vision and had achieved considerable success in different levels of vision tasks including image restoration. However, recently the Swin Transformer-based model also shows impressive performance, even surpasses the CNN-based methods to become the state-of-the-art on high-level vision tasks. In this paper, we proposed a restoration model called SUNet which uses the Swin Transformer layer as our basic block and…
Nanyang Technological University（南洋理工大学）；University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Image prior modeling is the key issue in image recovery, computational imaging, compresses sensing, and other inverse problems. Recent algorithms combining multiple effective priors such as the sparse or low-rank models, have demonstrated superior performance in various applications. However, the relationships among the popular image models are unclear, and no theory in general is available to demonstrate their connections. In this paper, we present a theoretical analysis on the image models, to bridge the gap between applications and image prior understanding, including sparsity, group-wise s…
Data61-Commonwealth Scientific Industrial Research Organization (CSIRO)（Data61-联邦科学与工业研究组织）；Australian National University（澳大利亚国立大学）
Deep convolutional neural networks perform better on images containing spatially invariant degradations, also known as synthetic degradations; however, their performance is limited on real-degraded photographs and requires multiple-stage network modeling. To advance the practicability of restoration algorithms, this paper proposes a novel single-stage blind real image restoration network (R$^2$Net) by employing a modular architecture. We use a residual on the residual structure to ease the flow of low-frequency information and apply feature attention to exploit the channel dependencies. Furthe…
Google Research（谷歌研究院）；Fondazione Human Technopole（人类技术城基金会）
Image denoising and artefact removal are complex inverse problems admitting multiple valid solutions. Unsupervised diversity restoration, that is, obtaining a diverse set of possible restorations given a corrupted image, is important for ambiguity removal in many applications such as microscopy where paired data for supervised training are often unobtainable. In real world applications, imaging noise and artefacts are typically hard to model, leading to unsatisfactory performance of existing unsupervised approaches. This work presents an interpretable approach for unsupervised and diverse imag…
Centre Suisse d’Electronique et de Microtechnique (CSEM)（瑞士电子与微技术中心（CSEM））；School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL）计算机与通信科学学院）
Plug-and-play denoisers can be used to perform generic image restoration tasks independent of the degradation type. These methods build on the fact that the Maximum a Posteriori (MAP) optimization can be solved using smaller sub-problems, including a MAP denoising optimization. We present the first end-to-end approach to MAP estimation for image denoising using deep neural networks. We show that our method is guaranteed to minimize the MAP denoising objective, which is then used in an optimization algorithm for generic image restoration. We provide theoretical analysis of our approach and show…
Due to the development of deep learning-based denoisers, the plug-and-play strategy has achieved great success in image restoration problems. However, existing plug-and-play image restoration methods are designed for non-blind Gaussian denoising such as zhang et al (2022), the performance of which visibly deteriorate for unknown noises. To push the limits of plug-and-play image restoration, we propose a novel framework with blind Gaussian prior, which can deal with more complicated image restoration problems in the real world. More specifically, we build up a new image restoration model by reg…
The principal rank-one (RO) components of an image represent the self-similarity of the image, which is an important property for image restoration. However, the RO components of a corrupted image could be decimated by the procedure of image denoising. We suggest that the RO property should be utilized and the decimation should be avoided in image restoration. To achieve this, we propose a new framework comprised of two modules, i.e., the RO decomposition and RO reconstruction. The RO decomposition is developed to decompose a corrupted image into the RO components and residual. This is achieve…
Denoising diffusion models for high-resolution microscopy image restoration
Frankfurt Institute for Advanced Studies (FIAS)（法兰克福高级研究所）；Goethe University Frankfurt（法兰克福大学）；Institute of Physical and Theoretical Chemistry（物理与理论化学研究所）
RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Paradigm Inc.（第四范式有限公司）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；LISMARS, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) restoration, which includes the tasks of denoising, compressed HSI reconstruction and inpainting. Unfortunately, while its restoration performance benefits from more spectral bands, its runtime also substantially increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch group should lie in this global low-rank subspace. This motivates us to propose a unified paradigm combining the spatial and …
Fraunhofer Institute for Computer Graphics Research (IGD)（弗劳恩霍夫计算机图形研究所）；University of Rostock（罗斯托克大学）
Visual inspection of underwater structures by vehicles, e.g. remotely operated vehicles (ROVs), plays an important role in scientific, military, and commercial sectors. However, the automatic extraction of information using software tools is hindered by the characteristics of water which degrade the quality of captured videos. As a contribution for restoring the color of underwater images, Underwater Denoising Autoencoder (UDAE) model is developed using a denoising autoencoder with U-Net architecture. The proposed network takes into consideration the accuracy and the computation cost to enable…
Since convolutional neural networks perform well in learning generalizable image priors from large-scale data, these models have been widely used in image denoising tasks. However, the computational complexity increases dramatically as well on complex model. In this paper, We propose a novel lightweight Complementary Attention Module, which includes a density module and a sparse module, which can cooperatively mine dense and sparse features for feature complementary learning to build an efficient lightweight architecture. Moreover, to reduce the loss of details caused by denoising, this paper …
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
The development of neural networks has greatly improved the performance in various computer vision tasks. In the filed of image denoising, convolutional neural network based methods such as DnCNN break through the limits of classical methods, achieving better quantitative results. However, the epistemic uncertainty existing in neural networks limits further improvements in their performance over denoising tasks. Therefore, we develop and study different solutions to minimize uncertainty and further improve the removal of noise. From the perspective of ensemble learning, we implement manipulati…
University of Chinese Academy of Sciences（中国科学院大学）；Research Center on Fictitious Economy and Data Science, Chinese Academy of Sciences（中国科学院虚拟经济与数据科学研究中心）；Key Laboratory of Big Data Mining and Knowledge Management Chinese Academy of Sciences（中国科学院大数据挖掘与知识管理重点实验室）
In this paper, we present a general framework for low-level vision tasks including image compression artifacts reduction and image denoising. Under this framework, a novel concatenated attention neural network (CANet) is specifically designed for image restoration. The main contributions of this paper are as follows: First, by applying concise but effective concatenation and feature selection mechanism, we establish a novel connection mechanism which connect different modules in the modules stacking network. Second, both pixel-wise and channel-wise attention mechanisms are used in each module …
School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）
Super-resolution and denoising are ill-posed yet fundamental image restoration tasks. In blind settings, the degradation kernel or the noise level are unknown. This makes restoration even more challenging, notably for learning-based methods, as they tend to overfit to the degradation seen during training. We present an analysis, in the frequency domain, of degradation-kernel overfitting in super-resolution and introduce a conditional learning perspective that extends to both super-resolution and denoising. Building on our formulation, we propose a stochastic frequency masking of images used in…
RELD: Regularization by Latent Diffusion Models for Image Restoration
University of Bologna（博洛尼亚大学）；University of Macerata（马切拉塔大学）；University of Modena and Reggio Emilia（摩德纳大学与雷焦艾米利亚大学）；University of Milan（米兰大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
Univ. Bordeaux（波尔多大学）；Bordeaux INP（波尔多国立理工学院）；CNRS（法国国家科学研究中心）；Université de Paris（巴黎大学）
In this work, we propose a framework to learn a local regularization model for solving general image restoration problems. This regularizer is defined with a fully convolutional neural network that sees the image through a receptive field corresponding to small image patches. The regularizer is then learned as a critic between unpaired distributions of clean and degraded patches using a Wasserstein generative adversarial networks based energy. This yields a regularization function that can be incorporated in any image restoration problem. The efficiency of the framework is finally shown on den…
Inception Institute of Artificial Intelligence（人工智能 inception 研究院）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）
The availability of large-scale datasets has helped unleash the true potential of deep convolutional neural networks (CNNs). However, for the single-image denoising problem, capturing a real dataset is an unacceptably expensive and cumbersome procedure. Consequently, image denoising algorithms are mostly developed and evaluated on synthetic data that is usually generated with a widespread assumption of additive white Gaussian noise (AWGN). While the CNNs achieve impressive results on these synthetic datasets, they do not perform well when applied on real camera images, as reported in recent be…
National Chung Hsing University（国立中兴大学）；National Taichung University of Science and Technology（国立台中科技大学）
Image restoration is a low-level vision task which is to restore degraded images to noise-free images. With the success of deep neural networks, the convolutional neural networks surpass the traditional restoration methods and become the mainstream in the computer vision area. To advance the performanceof denoising algorithms, we propose a blind real image denoising network (SRMNet) by employing a hierarchical architecture improved from U-Net. Specifically, we use a selective kernel with residual block on the hierarchical structure called M-Net to enrich the multi-scale semantic information. F…
Color image denoising is frequently encountered in various image processing and computer vision tasks. One traditional strategy is to convert the RGB image to a less correlated color space and denoise each channel of the new space separately. However, such a strategy can not fully exploit the correlated information between channels and is inadequate to obtain satisfactory results. To address this issue, this paper proposes a new multi-channel optimization model for color image denoising under the nuclear norm minus Frobenius norm minimization framework. Specifically, based on the block-matchin…
In general, image restoration involves mapping from low quality images to their high-quality counterparts. Such optimal mapping is usually non-linear and learnable by machine learning. Recently, deep convolutional neural networks have proven promising for such learning processing. It is desirable for an image processing network to support well with three vital tasks, namely, super-resolution, denoising, and deblocking. It is commonly recognized that these tasks have strong correlations. Therefore, it is imperative to harness the inter-task correlations. To this end, we propose the cross-scale …
We leverage deep learning techniques to jointly denoise and super-resolve biomedical images acquired with fluorescence microscopy. We develop a deep learning algorithm based on the networks and method described in the recent W2S paper to solve a joint denoising and super-resolution problem. Specifically, we address the restoration of SIM images from widefield images. Our TensorFlow model is trained on the W2S dataset of cell images and is made accessible online in this repository: https://github.com/mchatton/w2s-tensorflow. On test images, the model shows a visually-convincing denoising and in…
Deep neural network based methods are the state of the art in various image restoration problems. Standard supervised learning frameworks require a set of noisy measurement and clean image pairs for which a distance between the output of the restoration model and the ground truth, clean images is minimized. The ground truth images, however, are often unavailable or very expensive to acquire in real-world applications. We circumvent this problem by proposing a class of structured denoisers that can be decomposed as the sum of a nonlinear image-dependent mapping, a linear noise-dependent term an…
In image denoising (IDN) processing, the low-rank property is usually considered as an important image prior. As a convex relaxation approximation of low rank, nuclear norm based algorithms and their variants have attracted significant attention. These algorithms can be collectively called image domain based methods, whose common drawback is the requirement of great number of iterations for some acceptable solution. Meanwhile, the sparsity of images in a certain transform domain has also been exploited in image denoising problems. Sparsity transform learning algorithms can achieve extremely fa…
Universidad Politécnica de Madrid（马德里理工大学）；Centre national de la recherche scientifique (CNRS)（法国国家科学研究中心）；Interdisciplinary Institute for Neuroscience (IINS)（跨学科神经科学研究所）；University of Bordeaux（波尔多大学）；Mechanobiology Institute（机械生物学研究所）；National University of Singapore（新加坡国立大学）
Microscopy images often suffer from high levels of noise, which can hinder further analysis and interpretation. Content-aware image restoration (CARE) methods have been proposed to address this issue, but they often require large amounts of training data and suffer from over-fitting. To overcome these challenges, we propose a novel framework for few-shot microscopy image denoising. Our approach combines a generative adversarial network (GAN) trained via contrastive learning (CL) with two structure preserving loss terms (Structural Similarity Index and Total Variation loss) to further improve t…
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究所）；Wellcome Sanger Institute（惠康桑格研究所）；PerkinElmer Cellular Technologies（珀金埃尔默细胞技术公司）
Biomedical images are noisy. The imaging equipment itself has physical limitations, and the consequent experimental trade-offs between signal-to-noise ratio, acquisition speed, and imaging depth exacerbate the problem. Denoising is, therefore, an essential part of any image processing pipeline, and convolutional neural networks are currently the method of choice for this task. One popular approach, Noise2Noise, does not require clean ground truth, and instead, uses a second noisy copy as a training target. Self-supervised methods, like Noise2Self and Noise2Void, relax data requirements by lear…
Almost every single image restoration problem has a closely related parameter, such as the scale factor in super-resolution, the noise level in image denoising, and the quality factor in JPEG deblocking. Although recent studies on image restoration problems have achieved great success due to the development of deep neural networks, they handle the parameter involved in an unsophisticated way. Most previous researchers either treat problems with different parameter levels as independent tasks, and train a specific model for each parameter level; or simply ignore the parameter, and train a singl…
Total Variation (TV) and related extensions have been popular in image restoration due to their robust performance and wide applicability. While the original formulation is still relevant after two decades of extensive research, its extensions that combine derivatives of first- and second-order are now being explored for better performance, with examples being Combined Order TV (COTV) and Total Generalized Variation (TGV). As an improvement over such multi-order convex formulations, we propose a novel non-convex regularization functional which adaptively combines Hessian-Schatten (HS) norm and…
Adaptive image restoration models can restore images with different degradation levels at inference time without the need to retrain the model. We present an approach that is highly accurate and allows a significant reduction in the number of parameters. In contrast to existing methods, our approach can restore images using a single fixed-size model, regardless of the number of degradation levels. On popular datasets, our approach yields state-of-the-art results in terms of size and accuracy for a variety of image restoration tasks, including denoising, deJPEG, and super-resolution.
Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from low-quality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper, we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer. SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image recons…
In this paper the problem of image restoration (denoising and inpainting) is approached using sparse approximation of local image blocks. The local image blocks are extracted by sliding square windows over the image. An adaptive block size selection procedure for local sparse approximation is proposed, which affects the global recovery of underlying image. Ideally the adaptive local block selection yields the minimum mean square error (MMSE) in recovered image. This framework gives us a clustered image based on the selected block size, then each cluster is restored separately using sparse appr…
Argonne National Laboratory（阿贡国家实验室）；University of Chicago（芝加哥大学）
The use of deep learning models within scientific experimental facilities frequently requires low-latency inference, so that, for example, quality control operations can be performed while data are being collected. Edge computing devices can be useful in this context, as their low cost and compact form factor permit them to be co-located with the experimental apparatus. Can such devices, with their limited resources, can perform neural network feed-forward computations efficiently and effectively? We explore this question by evaluating the performance and accuracy of a scientific image restora…
City University of Hong Kong（香港城市大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）
Night imaging with modern smartphone cameras is troublesome due to low photon count and unavoidable noise in the imaging system. Directly adjusting exposure time and ISO ratings cannot obtain sharp and noise-free images at the same time in low-light conditions. Though many methods have been proposed to enhance noisy or blurry night images, their performances on real-world night photos are still unsatisfactory due to two main reasons: 1) Limited information in a single image and 2) Domain gap between synthetic training images and real-world photos (e.g., differences in blur area and resolution)…
Institut de Mathématiques de Bordeaux, CNRS, Université de Bordeaux, Bordeaux INP（波尔多数学研究所、法国国家科学研究中心、波尔多大学、波尔多国立理工学院）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Patch priors have become an important component of image restoration. A powerful approach in this category of restoration algorithms is the popular Expected Patch Log-Likelihood (EPLL) algorithm. EPLL uses a Gaussian mixture model (GMM) prior learned on clean image patches as a way to regularize degraded patches. In this paper, we show that a generalized Gaussian mixture model (GGMM) captures the underlying distribution of patches better than a GMM. Even though GGMM is a powerful prior to combine with EPLL, the non-Gaussianity of its components presents major challenges to be applied to a comp…
Center for Systems Biology Dresden（德累斯顿系统生物学中心）；Max-Planck Institute (CBG)（马克斯·普朗克分子细胞生物学与遗传学研究所）；School of Computer Science（计算机科学学院）；University of Birmingham（伯明翰大学）；Fondazione Human Technopole（人类技术基金会）
Deep Learning based methods have emerged as the indisputable leaders for virtually all image restoration tasks. Especially in the domain of microscopy images, various content-aware image restoration (CARE) approaches are now used to improve the interpretability of acquired data. Naturally, there are limitations to what can be restored in corrupted images, and like for all inverse problems, many potential solutions exist, and one of them must be chosen. Here, we propose DivNoising, a denoising approach based on fully convolutional variational autoencoders (VAEs), overcoming the problem of havin…
Plug-and-Play optimization recently emerged as a powerful technique for solving inverse problems by plugging a denoiser into a classical optimization algorithm. The denoiser accounts for the regularization and therefore implicitly determines the prior knowledge on the data, hence replacing typical handcrafted priors. In this paper, we extend the concept of plug-and-play optimization to use denoisers that can be parameterized for non-constant noise variance. In that aim, we introduce a preconditioning of the ADMM algorithm, which mathematically justifies the use of such an adjustable denoiser. …
The speckle phenomenon remains a major hurdle for the analysis of SAR images. The development of speckle reduction methods closely follows methodological progress in the field of image restoration. The advent of deep neural networks has offered new ways to tackle this longstanding problem. Deep learning for speckle reduction is a very active research topic and already shows restoration performances that exceed that of the previous generations of methods based on the concepts of patches, sparsity, wavelet transform or total variation minimization. The objective of this paper is to give an overv…
Modern applications such as self-driving cars and drones rely heavily upon robust object detection techniques. However, weather corruptions can hinder the object detectability and pose a serious threat to their navigation and reliability. Thus, there is a need for efficient denoising, deraining, and restoration techniques. Generative adversarial networks and transformers have been widely adopted for image restoration. However, the training of these methods is often unstable and time-consuming. Furthermore, when used for object detection (OD), the output images generated by these methods may pr…
Instituto de Telecomunicações（电信研究院）；Instituto Superior Técnico（高等技术学院）；Universidade de Lisboa（里斯本大学）
This paper introduces two very fast and competitive hyperspectral image (HSI) restoration algorithms: fast hyperspectral denoising (FastHyDe), a denoising algorithm able to cope with Gaussian and Poissonian noise, and fast hyperspectral inpainting (FastHyIn), an inpainting algorithm to restore HSIs where some observations from known pixels in some known bands are missing. FastHyDe and FastHyIn fully exploit extremely compact and sparse HSI representations linked with their low-rank and self-similarity characteristics. In a series of experiments with simulated and real data, the newly introduce…
