Blind Motion Deblurring with Cycle Generative Adversarial Networks
School of Automation, Nanjing University of Information Science and Technology（南京信息工程大学自动化学院）；B-DAT（B-DAT（注：此处B-DAT为原文缩写，无更多上下文时保留英文））；Nanjing University of Information Science and Technology（南京信息工程大学）
Quan Yuan, Junxia Li, Lingwei Zhang, Zhefu Wu, Guangyu Liu
Blind motion deblurring is one of the most basic and challenging problems in image processing and computer vision. It aims to recover a sharp image from its blurred version knowing nothing about the blur process. Many existing methods use Maximum A Posteriori (MAP) or Expectation Maximization (EM) frameworks to deal with this kind of problems, but they cannot handle well the figh frequency features of natural images. Most recently, deep neural networks have been emerging as a powerful tool for image deblurring. In this paper, we prove that encoder-decoder architecture gives better results for image deblurring tasks. In addition, we propose a novel end-to-end learning model which refines generative adversarial network by many novel training strategies so as to tackle the problem of deblurring. Experimental results show that our model can capture high frequency features well, and the results on benchmark dataset show that proposed model achieves the competitive performance.
Recently, end-to-end learning-based methods based on deep neural network (DNN) have been proven effective for blind deblurring. Without human-made assumptions and numerical algorithms, they are able to restore images with fewer artifacts and better perceptual quality. However, in practice, we also find some of their drawbacks. Without the theoretical guidance, these methods can not perform well when the motion is complex and sometimes generate unreasonable results. In this paper, for overcoming these drawbacks, we integrate deep convolution neural networks into conventional deblurring framewor…
In this paper, we introduce an end-to-end generative adversarial network (GAN) based on sparse learning for single image blind motion deblurring, which we called SL-CycleGAN. For the first time in blind motion deblurring, we propose a sparse ResNet-block as a combination of sparse convolution layers and a trainable spatial pooler k-winner based on HTM (Hierarchical Temporal Memory) to replace non-linearity such as ReLU in the ResNet-block of SL-CycleGAN generators. Furthermore, unlike many state-of-the-art GAN-based motion deblurring methods that treat motion deblurring as a linear end-to-end …
Beijing Jiaotong University（北京交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Xpeng Motors（小鹏汽车）
Non-uniform blur, mainly caused by camera shake and motions of multiple objects, is one of the most common causes of image quality degradation. However, the traditional blind deblurring methods based on blur kernel estimation do not perform well on complicated non-uniform motion blurs. Recent studies show that GAN-based approaches achieve impressive performance on deblurring tasks. In this letter, to further improve the performance of GAN-based methods on deblurring tasks, we propose an edge heuristic multi-scale generative adversarial network(GAN), which uses the "coarse-to-fine" scheme to re…
Australian National University（澳大利亚国立大学）；Tencent AI Laboratory（腾讯人工智能实验室）
Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and temporal domain (i.e., neighboring frames), and 2) how to restore sharp image details w.r.t. the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a DeBLuRring Network (DBLRNet) for spatial-temporal learning by applyin…
While working with fingerprint images acquired from crime scenes, mobile cameras, or low-quality sensors, it becomes difficult for automated identification systems to verify the identity due to image blur and distortion. We propose a fingerprint deblurring model FDeblur-GAN, based on the conditional Generative Adversarial Networks (cGANs) and multi-stage framework of the stack GAN. Additionally, we integrate two auxiliary sub-networks into the model for the deblurring task. The first sub-network is a ridge extractor model. It is added to generate ridge maps to ensure that fingerprint informati…
Kalinga Institute of Industrial Technology（卡林加工业技术学院）；Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
Deblurring is the task of restoring a blurred image to a sharp one, retrieving the information lost due to the blur. In blind deblurring we have no information regarding the blur kernel. As deblurring can be considered as an image to image translation task, deep learning based solutions, including the ones which use GAN (Generative Adversarial Network), have been proven effective for deblurring. Most of them have an encoder-decoder structure. Our objective is to try different GAN structures and improve its performance through various modifications to the existing structure for supervised deblu…
In this paper, we present a Fast Motion Deblurring-Conditional Generative Adversarial Network (FMD-cGAN) that helps in blind motion deblurring of a single image. FMD-cGAN delivers impressive structural similarity and visual appearance after deblurring an image. Like other deep neural network architectures, GANs also suffer from large model size (parameters) and computations. It is not easy to deploy the model on resource constraint devices such as mobile and robotics. With the help of MobileNet based architecture that consists of depthwise separable convolution, we reduce the model size and in…
School of Informatics（信息学院）；Xiamen University（厦门大学）
Low-quality face image restoration is a popular research direction in today's computer vision field. It can be used as a pre-work for tasks such as face detection and face recognition. At present, there is a lot of work to solve the problem of low-quality faces under various environmental conditions. This paper mainly focuses on the restoration of motion-blurred faces. In increasingly abundant mobile scenes, the fast recovery of motion-blurred faces can bring highly effective speed improvements in tasks such as face matching. In order to achieve this goal, a deblurring method for motion-blurre…
Facial image super-resolution (SR) is an important preprocessing for facial image analysis, face recognition, and image-based 3D face reconstruction. Recent convolutional neural network (CNN) based method has shown excellent performance by learning mapping relation using pairs of low-resolution (LR) and high-resolution (HR) facial images. However, since the HR facial image reconstruction using CNN is conventionally aimed to increase the PSNR and SSIM metrics, the reconstructed HR image might not be realistic even with high scores. An adversarial framework is proposed in this study to reconstru…
This thesis analyzes the challenging problem of Image Deblurring based on classical theorems and state-of-art methods proposed in recent years. By spectral analysis we mathematically show the effective of spectral regularization methods, and point out the linking between the spectral filtering result and the solution of the regularization optimization objective. For ill-posed problems like image deblurring, the optimization objective contains a regularization term (also called the regularization functional) that encodes our prior knowledge into the solution. We demonstrate how to craft a regul…
Northwestern Polytechnical University（西北工业大学）；The University of Adelaide（阿德莱德大学）
Blind image deblurring is a fundamental and challenging computer vision problem, which aims to recover both the blur kernel and the latent sharp image from only a blurry observation. Despite the superiority of deep learning methods in image deblurring have displayed, there still exists major challenge with various non-uniform motion blur. Previous methods simply take all the image features as the input to the decoder, which handles different degrees (e.g. large blur, small blur) simultaneously, leading to challenges for sharp image generation. To tackle the above problems, we present a deep tw…
Learning Deblurring Texture Prior from Unpaired Data with Diffusion Model
Xi’an Jiaotong University（西安交通大学）；Wuhan University（武汉大学）；Insta360（影石）；Nanjing University of Science and Technology（南京理工大学）；University of California, Merced（加利福尼亚大学默塞德分校）
This is a review on blind image deblurring. First, we formulate the blind image deblurring problem and explain why it is challenging. Next, we bring some psychological and cognitive studies on the way our human vision system deblurs. Then, relying on several previous reviews, we discuss the topic of metrics and datasets, which is non-trivial to blind deblurring. Finally, we introduce some typical optimization-based methods and learning-based methods.
Northwestern Polytechnical University（西北工业大学）；University of Technology Sydney（悉尼科技大学）
Single image blind deblurring is highly ill-posed as neither the latent sharp image nor the blur kernel is known. Even though considerable progress has been made, several major difficulties remain for blind deblurring, including the trade-off between high-performance deblurring and real-time processing. Besides, we observe that current single image blind deblurring networks cannot further improve or stabilize the performance but significantly degrades the performance when re-deblurring is repeatedly applied. This implies the limitation of these networks in modeling an ideal deblurring process.…
Due to image blurring image deconvolution is often used for studying biological structures in fluorescence microscopy. Fluorescence microscopy image volumes inherently suffer from intensity inhomogeneity, blur, and are corrupted by various types of noise which exacerbate image quality at deeper tissue depth. Therefore, quantitative analysis of fluorescence microscopy in deeper tissue still remains a challenge. This paper presents a three dimensional blind image deconvolution method for fluorescence microscopy using 3-way spatially constrained cycle-consistent adversarial networks. The restored…
Fast and easy blind deblurring using an inverse filter and PROBE
Tel Aviv University（特拉维夫大学）
PROBE（Progressive Removal of Blur Residual）是一个递归的盲去模糊框架。使用基本的修改后的反滤波器在其核心，PROBE的实验性能达到或超过了最先进的，无论是在视觉上和定量。值得注意的是，PROBE有助于揭示其收敛特性的分析。PROBE的动机是最近的想法渐进盲去模糊，但打破了以前的研究，其简单性，速度，性能和分析潜力。PROBE既不是函数最小化方法，也不是开环顺序方法（模糊核估计，然后是非盲去模糊）。PROBE是一种反馈方案，其独特的优势来自闭环架构，而不是其算法组件的准确性。
Faculty of Computer science, University of New Brunswick（新 Brunswick大学计算机科学学院）；Department of Computer Science, Brandon University（布兰顿大学计算机科学系）；Department of Applied Computer Science, University of Winnipeg（温尼伯大学应用计算机科学系）
Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
The goal of this project is to build a deep-learning solution that deblurs cornea scans, used for medical examination. The spherical shape of the eye prevents ophtamologist from having completely sharp image. Provided with a stack of corneas from confocal images, our approach is to build a model that performs an upscaling of the images using an SR (Super Resolution) Network.
Huazhong University of Science and Technology（华中科技大学）；University of California, Merced（加州大学默塞德分校）；Nanjing University of Science and Technology（南京理工大学）
We present an effective blind image deblurring method based on a data-driven discriminative prior.Our work is motivated by the fact that a good image prior should favor clear images over blurred images.In this work, we formulate the image prior as a binary classifier which can be achieved by a deep convolutional neural network (CNN).The learned prior is able to distinguish whether an input image is clear or not.Embedded into the maximum a posterior (MAP) framework, it helps blind deblurring in various scenarios, including natural, face, text, and low-illumination images.However, it is difficul…
We present a new end-to-end generative adversarial network (GAN) for single image motion deblurring, named DeblurGAN-v2, which considerably boosts state-of-the-art deblurring efficiency, quality, and flexibility. DeblurGAN-v2 is based on a relativistic conditional GAN with a double-scale discriminator. For the first time, we introduce the Feature Pyramid Network into deblurring, as a core building block in the generator of DeblurGAN-v2. It can flexibly work with a wide range of backbones, to navigate the balance between performance and efficiency. The plug-in of sophisticated backbones (e.g., …
Blind image deblurring is a particularly challenging inverse problem where the blur kernel is unknown and must be estimated en route to recover the deblurred image. The problem is of strong practical relevance since many imaging devices such as cellphone cameras, must rely on deblurring algorithms to yield satisfactory image quality. Despite significant research effort, handling large motions remains an open problem. In this paper, we develop a new method called Blind Image Deblurring using Row-Column Sparsity (BD-RCS) to address this issue. Specifically, we model the outer product of kernel a…
Peking University（北京大学）；National Institute of Informatics（国立信息学研究所）；Harbin Institute of Technology（哈尔滨工业大学）
Blind image deblurring, i.e., deblurring without knowledge of the blur kernel, is a highly ill-posed problem. The problem can be solved in two parts: i) estimate a blur kernel from the blurry image, and ii) given estimated blur kernel, de-convolve blurry input to restore the target image. In this paper, we propose a graph-based blind image deblurring algorithm by interpreting an image patch as a signal on a weighted graph. Specifically, we first argue that a skeleton image---a proxy that retains the strong gradients of the target but smooths out the details---can be used to accurately estimate…
Indian Institute of Technology Gandhinagar（甘地纳加尔印度理工学院）
Blind motion deblurring involves reconstructing a sharp image from an observation that is blurry. It is a problem that is ill-posed and lies in the categories of image restoration problems. The training data-based methods for image deblurring mostly involve training models that take a lot of time. These models are data-hungry i.e., they require a lot of training data to generate satisfactory results. Recently, there are various image feature learning methods developed which relieve us of the need for training data and perform image restoration and image synthesis, e.g., DIP, InGAN, and SinGAN.…
We present a novel, blind, single image deblurring method that utilizes information regarding blur kernels. Our model solves the deblurring problem by dividing it into two successive tasks: (1) blur kernel estimation and (2) sharp image restoration. We first introduce a kernel estimation network that produces adaptive blur kernels based on the analysis of the blurred image. The network learns the blur pattern of the input image and trains to generate the estimation of image-specific blur kernels. Subsequently, we propose a deblurring network that restores sharp images using the estimated blur …
When sailing at sea, the smart ship will inevitably produce swaying motion due to the action of wind, wave and current, which makes the image collected by the visual sensor appear motion blur. This will have an adverse effect on the object detection algorithm based on the vision sensor, thereby affect the navigation safety of the smart ship. In order to remove the motion blur in the images during the navigation of the smart ship, we propose SharpGAN, a new image deblurring method based on the generative adversarial network. First of all, the Receptive Field Block Net (RFBNet) is introduced to …
Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）；National Institute of Informatics（国立信息学研究所）
Blind image deblurring, i.e., deblurring without knowledge of the blur kernel, is a highly ill-posed problem. The problem can be solved in two parts: i) estimate a blur kernel from the blurry image, and ii) given estimated blur kernel, de-convolve blurry input to restore the target image. In this paper, by interpreting an image patch as a signal on a weighted graph, we first argue that a skeleton image---a proxy that retains the strong gradients of the target but smooths out the details---can be used to accurately estimate the blur kernel and has a unique bi-modal edge weight distribution. We …
Ukrainian Catholic University（乌克兰天主教大学）；ELEKS Ltd.（ELEKS有限公司）；Visual Recognition Group, Center for Machine Perception, FEE, CTU in Prague（布拉格捷克技术大学（CTU）费尔学院（FEE）机器感知中心视觉识别小组）
We present DeblurGAN, an end-to-end learned method for motion deblurring. The learning is based on a conditional GAN and the content loss . DeblurGAN achieves state-of-the art performance both in the structural similarity measure and visual appearance. The quality of the deblurring model is also evaluated in a novel way on a real-world problem -- object detection on (de-)blurred images. The method is 5 times faster than the closest competitor -- DeepDeblur. We also introduce a novel method for generating synthetic motion blurred images from sharp ones, allowing realistic dataset augmentation. …
Deep learning in motion deblurring: current status, benchmarks and future prospects
Institute of Systems Engineering, Academy of Military Science（军事科学院系统工程研究所）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）
Centre for Quantum Computation & Intelligent Systems（量子计算与智能系统中心）；Faculty of Engineering & Information Technology（工程与信息技术学院）；University of Technology Sydney（悉尼科技大学）
INRIA（法国国家信息与自动化研究所）；Département d’informatique de l’ENS（巴黎高等师范学院计算机科学系）；ENS（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL University（巴黎文理研究大学）；Xi’an Jiaotong University（西安交通大学）
Non-blind image deblurring is typically formulated as a linear least-squares problem regularized by natural priors on the corresponding sharp picture's gradients, which can be solved, for example, using a half-quadratic splitting method with Richardson fixed-point iterations for its least-squares updates and a proximal operator for the auxiliary variable updates. We propose to precondition the Richardson solver using approximate inverse filters of the (known) blur and natural image prior kernels. Using convolutions instead of a generic linear preconditioner allows extremely efficient parameter…
Diffusion model-based inverse problem solvers have demonstrated state-of-the-art performance in cases where the forward operator is known (i.e. non-blind). However, the applicability of the method to blind inverse problems has yet to be explored. In this work, we show that we can indeed solve a family of blind inverse problems by constructing another diffusion prior for the forward operator. Specifically, parallel reverse diffusion guided by gradients from the intermediate stages enables joint optimization of both the forward operator parameters as well as the image, such that both are jointly…
Multi-scale Frequency Enhancement Network for Blind Image Deblurring
The Institute of Systems Engineering, Academy of Military Science（军事科学院系统工程研究所）；School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机学院）；Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, Jiangnan University（江南大学江苏省模式识别与计算智能工程实验室）；College of Artificial Intelligence, China University of Petroleum(beijing)（中国石油大学（北京）人工智能学院）；Beijing Key Laboratory of Petroleum Data Mining, China University of Petroleum(beijing)（中国石油大学（北京）北京市石油数据挖掘重点实验室）
Seoul National University（首尔大学）；Inha University（仁荷大学）
Removing camera motion blur from a single light field is a challenging task since it is highly ill-posed inverse problem. The problem becomes even worse when blur kernel varies spatially due to scene depth variation and high-order camera motion. In this paper, we propose a novel algorithm to estimate all blur model variables jointly, including latent sub-aperture image, camera motion, and scene depth from the blurred 4D light field. Exploiting multi-view nature of a light field relieves the inverse property of the optimization by utilizing strong depth cues and multi-view blur observation. The…
The goal of blind image deblurring is to recover a sharp image from a motion blurred one without knowing the camera motion. Current state-of-the-art methods have a remarkably good performance on images with no noise or very low noise levels. However, the noiseless assumption is not realistic considering that low light conditions are the main reason for the presence of motion blur due to requiring longer exposure times. In fact, motion blur and high to moderate noise often appear together. Most works approach this problem by first estimating the blur kernel $k$ and then deconvolving the noisy b…
Blind Image deblurring tries to estimate blurriness and a latent image out of a blurred image. This estimation, as being an ill-posed problem, requires imposing restrictions on the latent image or a blur kernel that represents blurriness. Different from recent studies that impose some priors on the latent image, this paper regulates the structure of the blur kernel. We propose a kernel mixture structure while using the Gaussian kernel as a base kernel. By combining multiple Gaussian kernels structurally enhanced in terms of scales and centers, the kernel mixture becomes capable of modeling nea…
In this paper, we address the problem of blind deblurring with high efficiency. We propose a set of lightweight deep-wiener-network to finish the task with real-time speed. The Network contains a deep neural network for estimating parameters of wiener networks and a wiener network for deblurring. Experimental evaluations show that our approaches have an edge on State of the Art in terms of inference times and numbers of parameters. Two of our models can reach a speed of 100 images per second, which is qualified for real-time deblurring. Further research may focus on some real-world application…
Many deep learning based methods are designed to remove non-uniform (spatially variant) motion blur caused by object motion and camera shake without knowing the blur kernel. Some methods directly output the latent sharp image in one stage, while others utilize a multi-stage strategy (\eg multi-scale, multi-patch, or multi-temporal) to gradually restore the sharp image. However, these methods have the following two main issues: 1) The computational cost of multi-stage is high; 2) The same convolution kernel is applied in different regions, which is not an ideal choice for non-uniform blur. Henc…
Dalian University of Technology（大连理工大学）；Guilin University of Electronic Technology（桂林电子科技大学）
Blind image deblurring plays a very important role in many vision and multimedia applications. Most existing works tend to introduce complex priors to estimate the sharp image structures for blur kernel estimation. However, it has been verified that directly optimizing these models is challenging and easy to fall into degenerate solutions. Although several experience-based heuristic inference strategies, including trained networks and designed iterations, have been developed, it is still hard to obtain theoretically guaranteed accurate solutions. In this work, a collaborative learning framewor…
Hebei University of Technology（河北工业大学）；Australian National University（澳大利亚国立大学）；Tencent（腾讯）
Single-image super-resolution (SR) and multi-frame SR are two ways to super resolve low-resolution images. Single-Image SR generally handles each image independently, but ignores the temporal information implied in continuing frames. Multi-frame SR is able to model the temporal dependency via capturing motion information. However, it relies on neighbouring frames which are not always available in the real world. Meanwhile, slight camera shake easily causes heavy motion blur on long-distance-shot low-resolution images. To address these problems, a Blind Motion Deblurring Super-Reslution Network…
Image blurring refers to the degradation of an image wherein the image's overall sharpness decreases. Image blurring is caused by several factors. Additionally, during the image acquisition process, noise may get added to the image. Such a noisy and blurred image can be represented as the image resulting from the convolution of the original image with the associated point spread function, along with additive noise. However, the blurred image often contains inadequate information to uniquely determine the plausible original image. Based on the availability of blurring information, image deblurr…
