Image restoration has seen great progress in the last years thanks to the advances in deep neural networks. Most of these existing techniques are trained using full supervision with suitable image pairs to tackle a specific degradation. However, in a blind setting with unknown degradations this is not possible and a good prior remains crucial. Recently, neural network based approaches have been proposed to model such priors by leveraging either denoising autoencoders or the implicit regularization captured by the neural network structure itself. In contrast to this, we propose using normalizing flows to model the distribution of the target content and to use this as a prior in a maximum a posteriori (MAP) formulation. By expressing the MAP optimization process in the latent space through the learned bijective mapping, we are able to obtain solutions through gradient descent. To the best of our knowledge, this is the first work that explores normalizing flows as prior in image enhancement problems. Furthermore, we present experimental results for a number of different degradations on data sets varying in complexity and show competitive results when comparing with the deep image prior approach.
Due to the development of deep learning-based denoisers, the plug-and-play strategy has achieved great success in image restoration problems. However, existing plug-and-play image restoration methods are designed for non-blind Gaussian denoising such as zhang et al (2022), the performance of which visibly deteriorate for unknown noises. To push the limits of plug-and-play image restoration, we propose a novel framework with blind Gaussian prior, which can deal with more complicated image restoration problems in the real world. More specifically, we build up a new image restoration model by reg…
Blind super-resolution (SR) aims to recover high-quality visual textures from a low-resolution (LR) image, which is usually degraded by down-sampling blur kernels and additive noises. This task is extremely difficult due to the challenges of complicated image degradations in the real-world. Existing SR approaches either assume a predefined blur kernel or a fixed noise, which limits these approaches in challenging cases. In this paper, we propose a Degradation-guided Meta-restoration network for blind Super-Resolution (DMSR) that facilitates image restoration for real cases. DMSR consists of a …
Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）
Blind image restoration processors based on convolutional neural network (CNN) are intensively researched because of their high performance. However, they are too sensitive to the perturbation of the degradation model. They easily fail to restore the image whose degradation model is slightly different from the trained degradation model. In this paper, we propose a non-blind CNN-based image restoration processor, aiming to be robust against a perturbation of the degradation model compared to the blind restoration processor. Experimental comparisons demonstrate that the proposed non-blind CNN-ba…
Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Image quality is the basis of image communication and understanding tasks. Due to the blur and noise effects caused by imaging, transmission and other processes, the image quality is degraded. Blind image restoration is widely used to improve image quality, where the main goal is to faithfully estimate the blur kernel and the latent sharp image. In this study, based on experimental observation and research, an adaptively sparse regularized minimization method is originally proposed. The high-order gradients combine with low-order ones to form a hybrid regularization term, and an adaptive opera…
Deep Variational Network Toward Blind Image Restoration
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算学系）；Macau Institute of Systems Engineering, Macau University of Science and Technology（澳门科技大学澳门系统工程研究所）
Many image restoration techniques are highly dependent on the degradation used during training, and their performance declines significantly when applied to slightly different input. Blind and universal techniques attempt to mitigate this by producing a trained model that can adapt to varying conditions. However, blind techniques to date require prior knowledge of the degradation process, and assumptions regarding its parameter-space. In this paper we present the Self-Normalization Side-Chain (SCNC), a novel approach to blind universal restoration in which no prior knowledge of the degradation…
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Seoul National University（首尔国立大学）；Samsung Electronics Co., Ltd.（三星电子株式会社）
This paper presents a new variational inference framework for image restoration and a convolutional neural network (CNN) structure that can solve the restoration problems described by the proposed framework. Earlier CNN-based image restoration methods primarily focused on network architecture design or training strategy with non-blind scenarios where the degradation models are known or assumed. For a step closer to real-world applications, CNNs are also blindly trained with the whole dataset, including diverse degradations. However, the conditional distribution of a high-quality image given a …
Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
Beijing University of Posts and Telecommunications（北京邮电大学）；ETH Zürich（苏黎世联邦理工学院）；Shanghai University（上海大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；National Tsing Hua University（国立清华大学）
Institute of Information Science, Academia Sinica（中央研究院信息科学研究所）
Blind image restoration is a non-convex problem which involves restoration of images from an unknown blur kernel. The factors affecting the performance of this restoration are how much prior information about an image and a blur kernel are provided and what algorithm is used to perform the restoration task. Prior information on images is often employed to restore the sharpness of the edges of an image. By contrast, no consensus is still present regarding what prior information to use in restoring from a blur kernel due to complex image blurring processes. In this paper, we propose modelling of…
Blind Restoration of High-Resolution Ultrasound Video
Department of Mathematics, City University of Hong Kong（香港城市大学数学系）；Hong Kong Centre for Cerebro-cardiovascular Health Engineering（香港脑心血管健康工程中心）；Department of the Arts, University of Bologna（博洛尼亚大学艺术系）；Department of Computer Science and Engineering, University of Bologna（博洛尼亚大学计算机科学与工程系）；Department of Operations and Risk Management and School of Data Science, Lingnan University（岭南大学运营与风险管理系及数据科学学院）
State-of-the-art atmospheric turbulence image restoration methods utilize standard image processing tools such as optical flow, lucky region and blind deconvolution to restore the images. While promising results have been reported over the past decade, many of the methods are agnostic to the physical model that generates the distortion. In this paper, we revisit the turbulence restoration problem by analyzing the reference frame generation and the blind deconvolution steps in a typical restoration pipeline. By leveraging tools in large deviation theory, we rigorously prove the minimum number o…
BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration
Chair of Media Technology, Munich Institute of Robotics and Machine Intelligence（媒体技术教授会，慕尼黑机器人与机器智能研究所）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）
Versatile Cataract Fundus Image Restoration Model Utilizing Unpaired Cataract and High-quality Images
Shenzhen-Platform and Carrier (International Science and Technology Information Center) & Shenzhen Bay Lab（深圳平台与载体（国际科技信息中心）及深圳湾实验室）；Capital Medical University（首都医科大学）；Beijing Tongren Hospital（北京同仁医院）
Analysis and Benchmarking of Extending Blind Face Image Restoration to Videos
The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；The Guangdong University of Technology（广东工业大学）；Texas A&M University（德克萨斯农工大学）
National University of Singapore（新加坡国立大学）；Wuhan Cobot Technology Co., Ltd.（武汉科博特科技有限公司）；Huazhong University of Science and Technology（华中科技大学）
Unwanted nonlinear gamma distortion frequently occurs in a great diversity of images during the procedures of image acquisition, processing, and/or display. And the gamma distortion often varies with capture setup change and luminance variation. Blind inverse gamma correction, which automatically determines a proper restoration gamma value from a given image, is of paramount importance to attenuate the distortion. For blind inverse gamma correction, an adaptive gamma transformation method (AGT-ME) is proposed directly from a maximized differential entropy model. And the corresponding optimizat…
School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）
Super-resolution and denoising are ill-posed yet fundamental image restoration tasks. In blind settings, the degradation kernel or the noise level are unknown. This makes restoration even more challenging, notably for learning-based methods, as they tend to overfit to the degradation seen during training. We present an analysis, in the frequency domain, of degradation-kernel overfitting in super-resolution and introduce a conditional learning perspective that extends to both super-resolution and denoising. Building on our formulation, we propose a stochastic frequency masking of images used in…
Leveraging Classic Deconvolution and Feature Extraction in Zero-Shot Image Restoration
Technical University of Munich（慕尼黑工业大学）；Helmholtz AI, Helmholtz Munich - German Research Center for Environmental Health（亥姆霍兹人工智能平台，亥姆霍兹慕尼黑中心——德国环境健康研究中心）；Georg-August-University Göttingen（哥廷根乔治-奥古斯特大学）；GEOMAR Helmholtz Centre for Ocean Research Kiel（基尔亥姆霍兹海洋研究中心）
University of Bern（伯尔尼大学）；University of Maryland, College Park（马里兰大学学院市分校）
In this paper we introduce a natural image prior that directly represents a Gaussian-smoothed version of the natural image distribution. We include our prior in a formulation of image restoration as a Bayes estimator that also allows us to solve noise-blind image restoration problems. We show that the gradient of our prior corresponds to the mean-shift vector on the natural image distribution. In addition, we learn the mean-shift vector field using denoising autoencoders, and use it in a gradient descent approach to perform Bayes risk minimization. We demonstrate competitive results for noise-…
University of Cambridge（剑桥大学）；Samsung AI Center, Cambridge（三星剑桥人工智能中心）；University of Warwick（华威大学）
Deep learning-based blind super-resolution (SR) methods have recently achieved unprecedented performance in upscaling frames with unknown degradation. These models are able to accurately estimate the unknown downscaling kernel from a given low-resolution (LR) image in order to leverage the kernel during restoration. Although these approaches have largely been successful, they are predominantly image-based and therefore do not exploit the temporal properties of the kernels across multiple video frames. In this paper, we investigated the temporal properties of the kernels and highlighted its imp…
ETH Zürich（苏黎世联邦理工学院）；KU Leuven（鲁汶大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）
Recent works on plug-and-play image restoration have shown that a denoiser can implicitly serve as the image prior for model-based methods to solve many inverse problems. Such a property induces considerable advantages for plug-and-play image restoration (e.g., integrating the flexibility of model-based method and effectiveness of learning-based methods) when the denoiser is discriminatively learned via deep convolutional neural network (CNN) with large modeling capacity. However, while deeper and larger CNN models are rapidly gaining popularity, existing plug-and-play image restoration hinder…
Deep neural network based methods are the state of the art in various image restoration problems. Standard supervised learning frameworks require a set of noisy measurement and clean image pairs for which a distance between the output of the restoration model and the ground truth, clean images is minimized. The ground truth images, however, are often unavailable or very expensive to acquire in real-world applications. We circumvent this problem by proposing a class of structured denoisers that can be decomposed as the sum of a nonlinear image-dependent mapping, a linear noise-dependent term an…
University of Bern（伯尔尼大学）；University of Maryland, College Park（马里兰大学学院市分校）
We propose to leverage denoising autoencoder networks as priors to address image restoration problems. We build on the key observation that the output of an optimal denoising autoencoder is a local mean of the true data density, and the autoencoder error (the difference between the output and input of the trained autoencoder) is a mean shift vector. We use the magnitude of this mean shift vector, that is, the distance to the local mean, as the negative log likelihood of our natural image prior. For image restoration, we maximize the likelihood using gradient descent by backpropagating the auto…
Supervised learning with a convolutional neural network is recognized as a powerful means of image restoration. However, most such methods have been designed for application to grayscale and/or color images; therefore, they have limited success when applied to hyperspectral image restoration. This is partially owing to large datasets being difficult to collect, and also the heavy computational load associated with the restoration of an image with many spectral bands. To address this difficulty, we propose a novel self-supervised learning strategy for application to hyperspectral image restorat…
Image restoration problems are typical ill-posed problems where the regularization term plays an important role. The regularization term learned via generative approaches is easy to transfer to various image restoration, but offers inferior restoration quality compared with that learned via discriminative approaches. On the contrary, the regularization term learned via discriminative approaches are usually trained for a specific image restoration problem, and fail in the problem for which it is not trained. To address this issue, we propose a generic diffusion process (genericDP) to handle mul…
Xidian University（西安电子科技大学）；West Virginia University（西弗吉尼亚大学）
State-of-the-art approaches toward image restoration can be classified into model-based and learning-based. The former - best represented by sparse coding techniques - strive to exploit intrinsic prior knowledge about the unknown high-resolution images; while the latter - popularized by recently developed deep learning techniques - leverage external image prior from some training dataset. It is natural to explore their middle ground and pursue a hybrid image prior capable of achieving the best in both worlds. In this paper, we propose a systematic approach of achieving this goal called Structu…
Indian Institute of Technology Gandhinagar（甘地纳加尔印度理工学院）
Blind motion deblurring involves reconstructing a sharp image from an observation that is blurry. It is a problem that is ill-posed and lies in the categories of image restoration problems. The training data-based methods for image deblurring mostly involve training models that take a lot of time. These models are data-hungry i.e., they require a lot of training data to generate satisfactory results. Recently, there are various image feature learning methods developed which relieve us of the need for training data and perform image restoration and image synthesis, e.g., DIP, InGAN, and SinGAN.…
Perceptual image restoration seeks for high-fidelity images that most likely degrade to given images. For better visual quality, previous work proposed to search for solutions within the natural image manifold, by exploiting the latent space of a generative model. However, the quality of generated images are only guaranteed when latent embedding lies close to the prior distribution. In this work, we propose to restrict the feasible region within the prior manifold. This is accomplished with a non-parametric metric for two distributions: the Maximum Mean Discrepancy (MMD). Moreover, we model th…
Nanjing University of Science and Technology（南京理工大学）；SenseTime Research（商汤科技研究院）
Existing video super-resolution (SR) algorithms usually assume that the blur kernels in the degradation process are known and do not model the blur kernels in the restoration. However, this assumption does not hold for video SR and usually leads to over-smoothed super-resolved images. In this paper, we propose a deep convolutional neural network (CNN) model to solve video SR by a blur kernel modeling approach. The proposed deep CNN model consists of motion blur estimation, motion estimation, and latent image restoration modules. The motion blur estimation module is used to provide reliable blu…
Framelet-Based Blind Image Restoration with Minimax Concave Regularization
College of Artificial Intelligence and Big Data, Chongqing Polytechnic University of Electronic Technology, Chongqing,China（人工智能与大数据学院，重庆电子工程职业大学，重庆，中国）；Department of Engineering Sciences, Faculty of Advanced Technologies, University of Mohaghegh Ardabili, Ardabil, Iran（工程科学系，先进科技学院，莫哈格赫阿德布利大学，阿德布尔，伊朗）
Nanyang Technological University（南洋理工大学）；Michigan State University（密歇根州立大学）
Image restoration schemes based on the pre-trained deep models have received great attention due to their unique flexibility for solving various inverse problems. In particular, the Plug-and-Play (PnP) framework is a popular and powerful tool that can integrate an off-the-shelf deep denoiser for different image restoration tasks with known observation models. However, obtaining the observation model that exactly matches the actual one can be challenging in practice. Thus, the PnP schemes with conventional deep denoisers may fail to generate satisfying results in some real-world image restorati…
Tianjin University（天津大学）；Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；University of California, Merced（加州大学默塞德分校）
Most existing non-blind restoration methods are based on the assumption that a precise degradation model is known. As the degradation process can only be partially known or inaccurately modeled, images may not be well restored. Rain streak removal and image deconvolution with inaccurate blur kernels are two representative examples of such tasks. For rain streak removal, although an input image can be decomposed into a scene layer and a rain streak layer, there exists no explicit formulation for modeling rain streaks and the composition with scene layer. For blind deconvolution, as estimation e…
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The Hong Kong Polytechnic University（香港理工大学）
In supervised image restoration tasks, one key issue is how to obtain the aligned high-quality (HQ) and low-quality (LQ) training image pairs. Unfortunately, such HQ-LQ training pairs are hard to capture in practice, and hard to synthesize due to the complex unknown degradation in the wild. While several sophisticated degradation models have been manually designed to synthesize LQ images from their HQ counterparts, the distribution gap between the synthesized and real-world LQ images remains large. We propose a new approach to synthesizing realistic image restoration training pairs using the e…
