Non-local self-similarity and sparsity principles have proven to be powerful priors for natural image modeling. We propose a novel differentiable relaxation of joint sparsity that exploits both principles and leads to a general framework for image restoration which is (1) trainable end to end, (2) fully interpretable, and (3) much more compact than competing deep learning architectures. We apply this approach to denoising, jpeg deblocking, and demosaicking, and show that, with as few as 100K parameters, its performance on several standard benchmarks is on par or better than state-of-the-art methods that may have an order of magnitude or more parameters.
Tremendous advances in image restoration tasks such as denoising and super-resolution have been achieved using neural networks. Such approaches generally employ very deep architectures, large number of parameters, large receptive fields and high nonlinear modeling capacity. In order to obtain efficient and fast image restoration networks one should improve upon the above mentioned requirements. In this paper we propose a novel activation function, the multi-bin trainable linear unit (MTLU), for increasing the nonlinear modeling capacity together with lighter and shallower networks. We validate…
Graz University of Technology（格拉茨技术大学）；AIT Austrian Institute of Technology GmbH（奥地利理工学院）
Image restoration is a long-standing problem in low-level computer vision with many interesting applications. We describe a flexible learning framework based on the concept of nonlinear reaction diffusion models for various image restoration problems. By embodying recent improvements in nonlinear diffusion models, we propose a dynamic nonlinear reaction diffusion model with time-dependent parameters (\ie, linear filters and influence functions). In contrast to previous nonlinear diffusion models, all the parameters, including the filters and the influence functions, are simultaneously learned …
Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth "clean" hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where d…
In the past decade, sparsity-driven regularization has led to significant improvements in image reconstruction. Traditional regularizers, such as total variation (TV), rely on analytical models of sparsity. However, increasingly the field is moving towards trainable models, inspired from deep learning. Deep image prior (DIP) is a recent regularization framework that uses a convolutional neural network (CNN) architecture without data-driven training. This paper extends the DIP framework by combining it with the traditional TV regularization. We show that the inclusion of TV leads to considerabl…
Image restoration problems are typical ill-posed problems where the regularization term plays an important role. The regularization term learned via generative approaches is easy to transfer to various image restoration, but offers inferior restoration quality compared with that learned via discriminative approaches. On the contrary, the regularization term learned via discriminative approaches are usually trained for a specific image restoration problem, and fail in the problem for which it is not trained. To address this issue, we propose a generic diffusion process (genericDP) to handle mul…
Restore, Assess, Repeat: A Unified Framework for Iterative Image Restoration
Samsung AI Center Cambridge（三星剑桥人工智能中心）；National Taiwan University（国立台湾大学）；Technical University of Iasi（雅西科技大学）；Chang Gung University（长庚大学）；University of Wurzburg（维尔茨堡大学）；Queen Mary University of London（伦敦玛丽女王大学）
Google Research（谷歌研究院）；Fondazione Human Technopole（人类技术城基金会）
Image denoising and artefact removal are complex inverse problems admitting multiple valid solutions. Unsupervised diversity restoration, that is, obtaining a diverse set of possible restorations given a corrupted image, is important for ambiguity removal in many applications such as microscopy where paired data for supervised training are often unobtainable. In real world applications, imaging noise and artefacts are typically hard to model, leading to unsatisfactory performance of existing unsupervised approaches. This work presents an interpretable approach for unsupervised and diverse imag…
Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Grenoble INP（格勒诺布尔理工学院）
We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems, and whose architectures are derived from an optimization algorithm. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end. The priors used in this presentation include variants of total variat…
Southwest Jiaotong University（西南交通大学）；Nanyang Technological University（南洋理工大学）；Tongji University（同济大学）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）
Deep neural networks (DNN) have achieved great success in image restoration. However, most DNN methods are designed as a black box, lacking transparency and interpretability. Although some methods are proposed to combine traditional optimization algorithms with DNN, they usually demand pre-defined degradation processes or handcrafted assumptions, making it difficult to deal with complex and real-world applications. In this paper, we propose a Deep Generalized Unfolding Network (DGUNet) for image restoration. Concretely, without loss of interpretability, we integrate a gradient estimation strat…
LoopMoEVR: Loop-Based Degradation-Aware Mixture-of-Experts for Unified UHD Video Restoration
Shandong Normal University（山东师范大学）；China Academy of Information and Communications Technology（中国信息通信研究院）；Central South University（中南大学）；Nanjing University of Science and Technology（南京理工大学）；Hebei University of Technology（河北工业大学）；Shandong University of Finance and Economics（山东财经大学）；National University of Defense Technology（国防科技大学）
DVANet: Degradation-aware Visual-prior Alignment Network for Image Restoration
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）；State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智慧城市物联网国家重点实验室）
Zhengzhou University（郑州大学）；Univ Lyon, Ens de Lyon, Univ Lyon 1, CNRS, Laboratoire de Physique（里昂大学，里昂高等师范学院，里昂第一大学，法国国家科学研究中心，物理实验室）
Image restoration remains a challenging task in image processing. Numerous methods tackle this problem, often solved by minimizing a non-smooth penalized co-log-likelihood function. Although the solution is easily interpretable with theoretic guarantees, its estimation relies on an optimization process that can take time. Considering the research effort in deep learning for image classification and segmentation, this class of methods offers a serious alternative to perform image restoration but stays challenging to solve inverse problems. In this work, we design a deep network, named DeepPDNet…
RPCANet++: Deep Interpretable Robust PCA for Sparse Object Segmentation
School of Information and Communication Engineering and the Laboratory of Imaging Detection and Intelligent Perception, University of Electronic Science and Technology of China（信息与通信工程学院和图像检测与智能感知实验室，电子科技大学）；PCA Lab, VCIP, College of Computer Science, Nankai University（PCA实验室、VCIP、计算机科学学院，南开大学）；Department of Automation, Tsinghua University（自动化学院，清华大学）
Recent developments in deep learning have revolutionized the paradigm of image restoration. However, its applications on real image denoising are still limited, due to its sensitivity to training data and the complex nature of real image noise. In this work, we combine the robustness merit of model-based approaches and the learning power of data-driven approaches for real image denoising. Specifically, by integrating graph Laplacian regularization as a trainable module into a deep learning framework, we are less susceptible to overfitting than pure CNN-based approaches, achieving higher robust…
Bridging the Gap Between Image Restoration and Navigational Safety in Hazy Conditions: A New Visibility Estimation Metric for Maritime Surveillance
Sanya Science and Education Innovation Park, Wuhan University of Technology, Sanya 572000, China（武汉理工大学三亚科学教育创新园，三亚572000，中国）；School of Navigation, Wuhan University of Technology, Wuhan 430063, China（武汉理工大学航海学院，武汉430063，中国）；State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China（武汉理工大学海洋技术与安全国家重点实验室，武汉430063，中国）；Department of Mechanics and Maritime Sciences, Chalmers University of Technology, Göteborg 41296, Sweden（楚科奇大学机械与航海科学系，哥德堡41296，瑞典）
Discriminative learning based on convolutional neural networks (CNNs) aims to perform image restoration by learning from training examples of noisy-clean image pairs. It has become the go-to methodology for tackling image restoration and has outperformed the traditional non-local class of methods. However, the top-performing networks are generally composed of many convolutional layers and hundreds of neurons, with trainable parameters in excess of several millions. We claim that this is due to the inherent linear nature of convolution-based transformation, which is inadequate for handling seve…
With the recent increase in intelligent CCTVs for visual surveillance, a new image degradation that integrates resolution conversion and synthetic rain models is required. For example, in heavy rain, face images captured by CCTV from a distance have significant deterioration in both visibility and resolution. Unlike traditional image degradation models (IDM), such as rain removal and superresolution, this study addresses a new IDM referred to as a scale-aware heavy rain model and proposes a method for restoring high-resolution face images (HR-FIs) from low-resolution heavy rain face images (LR…
Breaking Degradation Coupling: A Structural Entropy Guided Decoupled Framework and Benchmark for Infrared Enhancement
Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；Department of Biomedical Engineering, Hefei University of Technology（合肥工业大学生物医学工程系）；School of Mathematics and Statistics, Yunnan University（云南大学数学与统计学院）
Compositional-Degradation UAV Image Restoration: Conditional Decoupled MoE Network and A Benchmark
Key Laboratory of Intelligent Computing & Signal Processing (Anhui University), Ministry of Education（教育部智能计算与信号处理重点实验室（安徽大学））；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China（北京理工大学计算机科学与技术学院）；School of Computer Science, Wuhan University, Wuhan, Hubei, China（武汉大学计算机学院）；Zhongguancun Academy, Beijing, China（中关村学院）
University of Science and Technology Beijing（北京科技大学）；Graz University of Technology（格拉茨技术大学）
Speckle reduction is a prerequisite for many image processing tasks in synthetic aperture radar (SAR) images, as well as all coherent images. In recent years, predominant state-of-the-art approaches for despeckling are usually based on nonlocal methods which mainly concentrate on achieving utmost image restoration quality, with relatively low computational efficiency. Therefore, in this study we aim to propose an efficient despeckling model with both high computational efficiency and high recovery quality. To this end, we exploit a newly-developed trainable nonlinear reaction diffusion(TNRD) f…
Jiangxi Normal University（江西师范大学）；East China Normal University（华东师范大学）
Single image dehazing is a challenging ill-posed restoration problem. Various prior-based and learning-based methods have been proposed. Most of them follow a classic atmospheric scattering model which is an elegant simplified physical model based on the assumption of single-scattering and homogeneous atmospheric medium. The formulation of haze in realistic environment is more complicated. In this paper, we propose to take its essential mechanism as "black box", and focus on learning an input-adaptive trainable end-to-end dehazing model. An U-Net like encoder-decoder deep network via progressi…
Pengcheng Laboratory（鹏城实验室）；Dalian University of Technology（大连理工大学）
Deep learning for image super-resolution (SR) has been investigated by numerous researchers in recent years. Most of the works concentrate on effective block designs and improve the network representation but lack interpretation. There are also iterative optimization-inspired networks for image SR, which take the solution step as a whole without giving an explicit optimization step. This paper proposes an unfolding iterative shrinkage thresholding algorithm (ISTA) inspired network for interpretable image SR. Specifically, we analyze the problem of image SR and propose a solution based on the I…
Image inpainting aims at restoring missing region of corrupted images, which has many applications such as image restoration and object removal. However, current GAN-based inpainting models fail to explicitly consider the semantic consistency between restored images and original images. Forexample, given a male image with image region of one eye missing, current models may restore it with a female eye. This is due to the ambiguity of GAN-based inpainting models: these models can generate many possible restorations given a missing region. To address this limitation, our key insight is that sema…
This work addresses the problem of discovering non-linear interpretable paths in the latent space of pre-trained GANs in a model-agnostic manner. In the proposed method, the discovery is driven by a set of pairs of natural language sentences with contrasting semantics, named semantic dipoles, that serve as the limits of the interpretation that we require by the trainable latent paths to encode. By using the pre-trained CLIP encoder, the sentences are projected into the vision-language space, where they serve as dipoles, and where RBF-based warping functions define a set of non-linear direction…
