去噪、去模糊和超分辨率等多种低级视觉任务从RGB图像出发，进一步减少退化以提升质量。然而，由于Image Signal Processor (ISP)变换，在sRGB域中对退化进行建模十分复杂。尽管这一问题已为人所知，但文献中极少有方法直接处理传感器RAW图像。在本研究中，我们直接在RAW域中解决图像恢复问题。我们设计了一条新的现实退化流水线，用于训练深度盲RAW恢复模型。该流水线考虑了现实传感器噪声、运动模糊、相机抖动及其他常见退化。使用我们的流水线和多传感器数据训练的模型，能够成功降低噪声和模糊，并恢复不同相机拍摄的RAW图像中的细节。据我们所知，这是对RAW图像恢复最详尽的分析。代码可在https://github.com/mv-lab/AISP获取。
英文摘要：
Multiple low-vision tasks such as denoising, deblurring and super-resolution depart from RGB images and further reduce the degradations, improving the quality. However, modeling the degradations in the sRGB domain is complicated because of the Image Signal Processor (ISP) transformations. Despite of this known issue, very few methods in the literature work directly with sensor RAW images. In this work we tackle image restoration directly in the RAW domain. We design a new realistic degradation pipeline for training deep blind RAW restoration models. Our pipeline considers realistic sensor noise, motion blur, camera shake, and other common degradations. The models trained with our pipeline and data from multiple sensors, can successfully reduce noise and blur, and recover details in RAW images captured from different cameras. To the best of our knowledge, this is the most exhaustive analysis on RAW image restoration. Code available at https://github.com/mv-lab/AISP
Due to the development of deep learning-based denoisers, the plug-and-play strategy has achieved great success in image restoration problems. However, existing plug-and-play image restoration methods are designed for non-blind Gaussian denoising such as zhang et al (2022), the performance of which visibly deteriorate for unknown noises. To push the limits of plug-and-play image restoration, we propose a novel framework with blind Gaussian prior, which can deal with more complicated image restoration problems in the real world. More specifically, we build up a new image restoration model by reg…
Blind super-resolution (SR) aims to recover high-quality visual textures from a low-resolution (LR) image, which is usually degraded by down-sampling blur kernels and additive noises. This task is extremely difficult due to the challenges of complicated image degradations in the real-world. Existing SR approaches either assume a predefined blur kernel or a fixed noise, which limits these approaches in challenging cases. In this paper, we propose a Degradation-guided Meta-restoration network for blind Super-Resolution (DMSR) that facilitates image restoration for real cases. DMSR consists of a …
Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）
Blind image restoration processors based on convolutional neural network (CNN) are intensively researched because of their high performance. However, they are too sensitive to the perturbation of the degradation model. They easily fail to restore the image whose degradation model is slightly different from the trained degradation model. In this paper, we propose a non-blind CNN-based image restoration processor, aiming to be robust against a perturbation of the degradation model compared to the blind restoration processor. Experimental comparisons demonstrate that the proposed non-blind CNN-ba…
Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Image quality is the basis of image communication and understanding tasks. Due to the blur and noise effects caused by imaging, transmission and other processes, the image quality is degraded. Blind image restoration is widely used to improve image quality, where the main goal is to faithfully estimate the blur kernel and the latent sharp image. In this study, based on experimental observation and research, an adaptively sparse regularized minimization method is originally proposed. The high-order gradients combine with low-order ones to form a hybrid regularization term, and an adaptive opera…
Deep Variational Network Toward Blind Image Restoration
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算学系）；Macau Institute of Systems Engineering, Macau University of Science and Technology（澳门科技大学澳门系统工程研究所）
Many image restoration techniques are highly dependent on the degradation used during training, and their performance declines significantly when applied to slightly different input. Blind and universal techniques attempt to mitigate this by producing a trained model that can adapt to varying conditions. However, blind techniques to date require prior knowledge of the degradation process, and assumptions regarding its parameter-space. In this paper we present the Self-Normalization Side-Chain (SCNC), a novel approach to blind universal restoration in which no prior knowledge of the degradation…
Department of Computer Science（计算机科学系）；DisneyResearch|Studios（迪士尼研究院|工作室）；ETH Zurich（苏黎世联邦理工学院）
Image restoration has seen great progress in the last years thanks to the advances in deep neural networks. Most of these existing techniques are trained using full supervision with suitable image pairs to tackle a specific degradation. However, in a blind setting with unknown degradations this is not possible and a good prior remains crucial. Recently, neural network based approaches have been proposed to model such priors by leveraging either denoising autoencoders or the implicit regularization captured by the neural network structure itself. In contrast to this, we propose using normalizin…
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Low-light imaging with handheld mobile devices is a challenging issue. Limited by the existing models and training data, most existing methods cannot be effectively applied in real scenarios. In this paper, we propose a new low-light image restoration method by using the complementary information of short- and long-exposure images. We first propose a novel data generation method to synthesize realistic short- and longexposure raw images by simulating the imaging pipeline in lowlight environment. Then, we design a new long-short-exposure fusion network (LSFNet) to deal with the problems of low-…
Seoul National University（首尔国立大学）；Samsung Electronics Co., Ltd.（三星电子株式会社）
This paper presents a new variational inference framework for image restoration and a convolutional neural network (CNN) structure that can solve the restoration problems described by the proposed framework. Earlier CNN-based image restoration methods primarily focused on network architecture design or training strategy with non-blind scenarios where the degradation models are known or assumed. For a step closer to real-world applications, CNNs are also blindly trained with the whole dataset, including diverse degradations. However, the conditional distribution of a high-quality image given a …
Survey on Deep Face Restoration: From Non-blind to Blind and Beyond
Beijing University of Posts and Telecommunications（北京邮电大学）；ETH Zürich（苏黎世联邦理工学院）；Shanghai University（上海大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；National Tsing Hua University（国立清华大学）
Institute of Information Science, Academia Sinica（中央研究院信息科学研究所）
Blind image restoration is a non-convex problem which involves restoration of images from an unknown blur kernel. The factors affecting the performance of this restoration are how much prior information about an image and a blur kernel are provided and what algorithm is used to perform the restoration task. Prior information on images is often employed to restore the sharpness of the edges of an image. By contrast, no consensus is still present regarding what prior information to use in restoring from a blur kernel due to complex image blurring processes. In this paper, we propose modelling of…
Blind Restoration of High-Resolution Ultrasound Video
Department of Mathematics, City University of Hong Kong（香港城市大学数学系）；Hong Kong Centre for Cerebro-cardiovascular Health Engineering（香港脑心血管健康工程中心）；Department of the Arts, University of Bologna（博洛尼亚大学艺术系）；Department of Computer Science and Engineering, University of Bologna（博洛尼亚大学计算机科学与工程系）；Department of Operations and Risk Management and School of Data Science, Lingnan University（岭南大学运营与风险管理系及数据科学学院）
State-of-the-art atmospheric turbulence image restoration methods utilize standard image processing tools such as optical flow, lucky region and blind deconvolution to restore the images. While promising results have been reported over the past decade, many of the methods are agnostic to the physical model that generates the distortion. In this paper, we revisit the turbulence restoration problem by analyzing the reference frame generation and the blind deconvolution steps in a typical restoration pipeline. By leveraging tools in large deviation theory, we rigorously prove the minimum number o…
In order to get raw images of high quality for downstream Image Signal Process (ISP), in this paper we present an Efficient Locally Multiplicative Transformer called ELMformer for raw image restoration. ELMformer contains two core designs especially for raw images whose primitive attribute is single-channel. The first design is a Bi-directional Fusion Projection (BFP) module, where we consider both the color characteristics of raw images and spatial structure of single-channel. The second one is that we propose a Locally Multiplicative Self-Attention (L-MSA) scheme to effectively deliver infor…
BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration
Chair of Media Technology, Munich Institute of Robotics and Machine Intelligence（媒体技术教授会，慕尼黑机器人与机器智能研究所）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）
Versatile Cataract Fundus Image Restoration Model Utilizing Unpaired Cataract and High-quality Images
Shenzhen-Platform and Carrier (International Science and Technology Information Center) & Shenzhen Bay Lab（深圳平台与载体（国际科技信息中心）及深圳湾实验室）；Capital Medical University（首都医科大学）；Beijing Tongren Hospital（北京同仁医院）
Analysis and Benchmarking of Extending Blind Face Image Restoration to Videos
The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；The Guangdong University of Technology（广东工业大学）；Texas A&M University（德克萨斯农工大学）
YOND: Practical Blind Raw Image Denoising Free from Camera-Specific Data Dependency
School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；School of Artificial Intelligence, Beijing Normal University（人工智能学院，北京师范大学）
National University of Singapore（新加坡国立大学）；Wuhan Cobot Technology Co., Ltd.（武汉科博特科技有限公司）；Huazhong University of Science and Technology（华中科技大学）
Unwanted nonlinear gamma distortion frequently occurs in a great diversity of images during the procedures of image acquisition, processing, and/or display. And the gamma distortion often varies with capture setup change and luminance variation. Blind inverse gamma correction, which automatically determines a proper restoration gamma value from a given image, is of paramount importance to attenuate the distortion. For blind inverse gamma correction, an adaptive gamma transformation method (AGT-ME) is proposed directly from a maximized differential entropy model. And the corresponding optimizat…
Binarized High-Efficiency RAW Video Restoration and Beyond
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Chinese Aeronautical Establishment（中国航空研究院）；Westlake University（西湖大学）；Shanghai Jiao Tong University（上海交通大学）
School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）
Super-resolution and denoising are ill-posed yet fundamental image restoration tasks. In blind settings, the degradation kernel or the noise level are unknown. This makes restoration even more challenging, notably for learning-based methods, as they tend to overfit to the degradation seen during training. We present an analysis, in the frequency domain, of degradation-kernel overfitting in super-resolution and introduce a conditional learning perspective that extends to both super-resolution and denoising. Building on our formulation, we propose a stochastic frequency masking of images used in…
Leveraging Classic Deconvolution and Feature Extraction in Zero-Shot Image Restoration
Technical University of Munich（慕尼黑工业大学）；Helmholtz AI, Helmholtz Munich - German Research Center for Environmental Health（亥姆霍兹人工智能平台，亥姆霍兹慕尼黑中心——德国环境健康研究中心）；Georg-August-University Göttingen（哥廷根乔治-奥古斯特大学）；GEOMAR Helmholtz Centre for Ocean Research Kiel（基尔亥姆霍兹海洋研究中心）
University of Bern（伯尔尼大学）；University of Maryland, College Park（马里兰大学学院市分校）
In this paper we introduce a natural image prior that directly represents a Gaussian-smoothed version of the natural image distribution. We include our prior in a formulation of image restoration as a Bayes estimator that also allows us to solve noise-blind image restoration problems. We show that the gradient of our prior corresponds to the mean-shift vector on the natural image distribution. In addition, we learn the mean-shift vector field using denoising autoencoders, and use it in a gradient descent approach to perform Bayes risk minimization. We demonstrate competitive results for noise-…
University of Cambridge（剑桥大学）；Samsung AI Center, Cambridge（三星剑桥人工智能中心）；University of Warwick（华威大学）
Deep learning-based blind super-resolution (SR) methods have recently achieved unprecedented performance in upscaling frames with unknown degradation. These models are able to accurately estimate the unknown downscaling kernel from a given low-resolution (LR) image in order to leverage the kernel during restoration. Although these approaches have largely been successful, they are predominantly image-based and therefore do not exploit the temporal properties of the kernels across multiple video frames. In this paper, we investigated the temporal properties of the kernels and highlighted its imp…
Deep neural network based methods are the state of the art in various image restoration problems. Standard supervised learning frameworks require a set of noisy measurement and clean image pairs for which a distance between the output of the restoration model and the ground truth, clean images is minimized. The ground truth images, however, are often unavailable or very expensive to acquire in real-world applications. We circumvent this problem by proposing a class of structured denoisers that can be decomposed as the sum of a nonlinear image-dependent mapping, a linear noise-dependent term an…
University of Bern（伯尔尼大学）；University of Maryland, College Park（马里兰大学学院市分校）
We propose to leverage denoising autoencoder networks as priors to address image restoration problems. We build on the key observation that the output of an optimal denoising autoencoder is a local mean of the true data density, and the autoencoder error (the difference between the output and input of the trained autoencoder) is a mean shift vector. We use the magnitude of this mean shift vector, that is, the distance to the local mean, as the negative log likelihood of our natural image prior. For image restoration, we maximize the likelihood using gradient descent by backpropagating the auto…
Image restoration problems are typical ill-posed problems where the regularization term plays an important role. The regularization term learned via generative approaches is easy to transfer to various image restoration, but offers inferior restoration quality compared with that learned via discriminative approaches. On the contrary, the regularization term learned via discriminative approaches are usually trained for a specific image restoration problem, and fail in the problem for which it is not trained. To address this issue, we propose a generic diffusion process (genericDP) to handle mul…
Efficient RAW Image Deblurring with Adaptive Frequency Modulation
School of Mathematics, Tianjin University（天津大学数学学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Indian Institute of Technology Gandhinagar（甘地纳加尔印度理工学院）
Blind motion deblurring involves reconstructing a sharp image from an observation that is blurry. It is a problem that is ill-posed and lies in the categories of image restoration problems. The training data-based methods for image deblurring mostly involve training models that take a lot of time. These models are data-hungry i.e., they require a lot of training data to generate satisfactory results. Recently, there are various image feature learning methods developed which relieve us of the need for training data and perform image restoration and image synthesis, e.g., DIP, InGAN, and SinGAN.…
Nanjing University of Science and Technology（南京理工大学）；SenseTime Research（商汤科技研究院）
Existing video super-resolution (SR) algorithms usually assume that the blur kernels in the degradation process are known and do not model the blur kernels in the restoration. However, this assumption does not hold for video SR and usually leads to over-smoothed super-resolved images. In this paper, we propose a deep convolutional neural network (CNN) model to solve video SR by a blur kernel modeling approach. The proposed deep CNN model consists of motion blur estimation, motion estimation, and latent image restoration modules. The motion blur estimation module is used to provide reliable blu…
Framelet-Based Blind Image Restoration with Minimax Concave Regularization
College of Artificial Intelligence and Big Data, Chongqing Polytechnic University of Electronic Technology, Chongqing,China（人工智能与大数据学院，重庆电子工程职业大学，重庆，中国）；Department of Engineering Sciences, Faculty of Advanced Technologies, University of Mohaghegh Ardabili, Ardabil, Iran（工程科学系，先进科技学院，莫哈格赫阿德布利大学，阿德布尔，伊朗）
INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）
We address the problem of non-blind deblurring and demosaicking of noisy raw images. We adapt an existing learning-based approach to RGB image deblurring to handle raw images by introducing a new interpretable module that jointly demosaicks and deblurs them. We train this model on RGB images converted into raw ones following a realistic invertible camera pipeline. We demonstrate the effectiveness of this model over two-stage approaches stacking demosaicking and deblurring modules on quantitive benchmarks. We also apply our approach to remove a camera's inherent blur (its color-dependent point-…
Tianjin University（天津大学）；Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；University of California, Merced（加州大学默塞德分校）
Most existing non-blind restoration methods are based on the assumption that a precise degradation model is known. As the degradation process can only be partially known or inaccurately modeled, images may not be well restored. Rain streak removal and image deconvolution with inaccurate blur kernels are two representative examples of such tasks. For rain streak removal, although an input image can be decomposed into a scene layer and a rain streak layer, there exists no explicit formulation for modeling rain streaks and the composition with scene layer. For blind deconvolution, as estimation e…
