Currently, many blind deblurring methods assume blurred images are noise-free and perform unsatisfactorily on the blurry images with noise. Unfortunately, noise is quite common in real scenes. A straightforward solution is to denoise images before deblurring them. However, even state-of-the-art denoisers cannot guarantee to remove noise entirely. Slight residual noise in the denoised images could cause significant artifacts in the deblurring stage. To tackle this problem, we propose a cascaded framework consisting of a denoiser subnetwork and a deblurring subnetwork. In contrast to previous methods, we train the two subnetworks jointly. Joint learning reduces the effect of the residual noise after denoising on deblurring, hence improves the robustness of deblurring to heavy noise. Moreover, our method is also helpful for blur kernel estimation. Experiments on the CelebA dataset and the GOPRO dataset show that our method performs favorably against several state-of-the-art methods.
The conventional methods for estimating camera poses and scene structures from severely blurry or low resolution images often result in failure. The off-the-shelf deblurring or super-resolution methods may show visually pleasing results. However, applying each technique independently before matching is generally unprofitable because this naive series of procedures ignores the consistency between images. In this paper, we propose a pioneering unified framework that solves four problems simultaneously, namely, dense depth reconstruction, camera pose estimation, super-resolution, and deblurring. …
Motion blur is a common photography artifact in dynamic environments that typically comes jointly with the other types of degradation. This paper reviews the NTIRE 2021 Challenge on Image Deblurring. In this challenge report, we describe the challenge specifics and the evaluation results from the 2 competition tracks with the proposed solutions. While both the tracks aim to recover a high-quality clean image from a blurry image, different artifacts are jointly involved. In track 1, the blurry images are in a low resolution while track 2 images are compressed in JPEG format. In each competition…
Jointly RS Image Deblurring and Super-Resolution with Adjustable-Kernel and Multi-Domain Attention
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）
EndoCaver: Handling Fog, Blur and Glare in Endoscopic Images via Joint Deblurring-Segmentation
1 CyPhi ( ) AI Research Lab, School of IT, Monash University, Malaysia Campus；2 Department of Electronic \& Computer Engineering, The Hong Kong University of Science；3 Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Sensetime Research（商汤科技研究院）；Tencent（腾讯）；South China University of Technology（华南理工大学）；University of Science and Technology of China（中国科学技术大学）；University of California at Merced（加州大学默塞德分校）；Tencent AI Lab（腾讯人工智能实验室）；Nanjing University of Science and Technology（南京理工大学）
We address the problem of restoring a high-resolution face image from a blurry low-resolution input. This problem is difficult as super-resolution and deblurring need to be tackled simultaneously. Moreover, existing algorithms cannot handle face images well as low-resolution face images do not have much texture which is especially critical for deblurring. In this paper, we propose an effective algorithm by utilizing the domain-specific knowledge of human faces to recover high-quality faces. We first propose a facial component guided deep Convolutional Neural Network (CNN) to restore a coarse f…
Blind image deblurring is a particularly challenging inverse problem where the blur kernel is unknown and must be estimated en route to recover the deblurred image. The problem is of strong practical relevance since many imaging devices such as cellphone cameras, must rely on deblurring algorithms to yield satisfactory image quality. Despite significant research effort, handling large motions remains an open problem. In this paper, we develop a new method called Blind Image Deblurring using Row-Column Sparsity (BD-RCS) to address this issue. Specifically, we model the outer product of kernel a…
Night photography typically suffers from both low light and blurring issues due to the dim environment and the common use of long exposure. While existing light enhancement and deblurring methods could deal with each problem individually, a cascade of such methods cannot work harmoniously to cope well with joint degradation of visibility and textures. Training an end-to-end network is also infeasible as no paired data is available to characterize the coexistence of low light and blurs. We address the problem by introducing a novel data synthesis pipeline that models realistic low-light blurrin…
VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising
The Chinese University of Hong Kong（香港中文大学）；National University of Defense Technology（国防科技大学）；City University of Hong Kong（香港城市大学）；Nanjing University of Science and Technology（南京理工大学）
INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）
We address the problem of non-blind deblurring and demosaicking of noisy raw images. We adapt an existing learning-based approach to RGB image deblurring to handle raw images by introducing a new interpretable module that jointly demosaicks and deblurs them. We train this model on RGB images converted into raw ones following a realistic invertible camera pipeline. We demonstrate the effectiveness of this model over two-stage approaches stacking demosaicking and deblurring modules on quantitive benchmarks. We also apply our approach to remove a camera's inherent blur (its color-dependent point-…
Xi’an Jiaotong University（西安交通大学）；Hikvision Research（海康威视研究院）；University of California, Merced（加州大学默塞德分校）；Google Cloud（谷歌云）
Single-image super-resolution is a fundamental task for vision applications to enhance the image quality with respect to spatial resolution. If the input image contains degraded pixels, the artifacts caused by the degradation could be amplified by super-resolution methods. Image blur is a common degradation source. Images captured by moving or still cameras are inevitably affected by motion blur due to relative movements between sensors and objects. In this work, we focus on the super-resolution task with the presence of motion blur. We propose a deep gated fusion convolution neural network to…
Australian National University（澳大利亚国立大学）；Northwestern Polytechnical University（西北工业大学）；Australia Centre for Robotic Vision（澳大利亚机器人视觉中心）
Camera shake during exposure is a major problem in hand-held photography, as it causes image blur that destroys details in the captured images.~In the real world, such blur is mainly caused by both the camera motion and the complex scene structure.~While considerable existing approaches have been proposed based on various assumptions regarding the scene structure or the camera motion, few existing methods could handle the real 6 DoF camera motion.~In this paper, we propose to jointly estimate the 6 DoF camera motion and remove the non-uniform blur caused by camera motion by exploiting their un…
CUHK（香港中文大学）；NVIDIA AI Technology Center（英伟达人工智能技术中心）；SenseTime Research（商汤科技研究院）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）；Xidian University（西安电子科技大学）
In various learning-based image restoration tasks, such as image denoising and image super-resolution, the degradation representations were widely used to model the degradation process and handle complicated degradation patterns. However, they are less explored in learning-based image deblurring as blur kernel estimation cannot perform well in real-world challenging cases. We argue that it is particularly necessary for image deblurring to model degradation representations since blurry patterns typically show much larger variations than noisy patterns or high-frequency textures.In this paper, w…
This paper tackles the problem of dynamic scene deblurring. Although end-to-end fully convolutional designs have recently advanced the state-of-the-art in non-uniform motion deblurring, their performance-complexity trade-off is still sub-optimal. Existing approaches achieve a large receptive field by a simple increment in the number of generic convolution layers, kernel-size, which comes with the burden of the increase in model size and inference speed. In this work, we propose an efficient pixel adaptive and feature attentive design for handling large blur variations within and across differe…
Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Pontificia Universidad Católica del Perú（秘鲁天主教大学）
Two different approaches have recently been proposed for boundary handling in convolutional sparse representations, avoiding potential boundary artifacts arising from the circular boundary conditions implied by the use of frequency domain solution methods by introducing a spatial mask into the convolutional sparse coding problem. In the present paper we show that, under certain circumstances, these methods fail in their design goal of avoiding boundary artifacts. The reasons for this failure are discussed, a solution is proposed, and the practical implications are illustrated in an image deblu…
Efficient RAW Image Deblurring with Adaptive Frequency Modulation
School of Mathematics, Tianjin University（天津大学数学学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Non-uniform image deblurring is a challenging task due to the lack of temporal and textural information in the blurry image itself. Complementary information from auxiliary sensors such event sensors are being explored to address these limitations. The latter can record changes in a logarithmic intensity asynchronously, called events, with high temporal resolution and high dynamic range. Current event-based deblurring methods combine the blurry image with events to jointly estimate per-pixel motion and the deblur operator. In this paper, we argue that a divide-and-conquer approach is more suit…
Real-world dynamic scene deblurring has long been a challenging task since paired blurry-sharp training data is unavailable. Conventional Maximum A Posteriori estimation and deep learning-based deblurring methods are restricted by handcrafted priors and synthetic blurry-sharp training pairs respectively, thereby failing to generalize to real dynamic blurriness. To this end, we propose a Neural Maximum A Posteriori (NeurMAP) estimation framework for training neural networks to recover blind motion information and sharp content from unpaired data. The proposed NeruMAP consists of a motion estima…
City University of Hong Kong（香港城市大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）
Night imaging with modern smartphone cameras is troublesome due to low photon count and unavoidable noise in the imaging system. Directly adjusting exposure time and ISO ratings cannot obtain sharp and noise-free images at the same time in low-light conditions. Though many methods have been proposed to enhance noisy or blurry night images, their performances on real-world night photos are still unsatisfactory due to two main reasons: 1) Limited information in a single image and 2) Domain gap between synthetic training images and real-world photos (e.g., differences in blur area and resolution)…
Nanjing University of Science and Technology（南京理工大学）；CETC（中国电子科技集团公司）
We present an effective and efficient method that explores the properties of Transformers in the frequency domain for high-quality image deblurring. Our method is motivated by the convolution theorem that the correlation or convolution of two signals in the spatial domain is equivalent to an element-wise product of them in the frequency domain. This inspires us to develop an efficient frequency domain-based self-attention solver (FSAS) to estimate the scaled dot-product attention by an element-wise product operation instead of the matrix multiplication in the spatial domain. In addition, we no…
DeepRFTv2: Kernel-level Learning for Image Deblurring
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University（上海多维信息处理重点实验室，华东师范大学）；State Key Laboratory of Infrared Physics, Shanghai Institute of Technical Physics, Chinese Academy of Sciences（红外物理国家重点实验室，上海技术物理研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Blind image deblurring is an important yet very challenging problem in low-level vision. Traditional optimization based methods generally formulate this task as a maximum-a-posteriori estimation or variational inference problem, whose performance highly relies on the handcraft priors for both the latent image and the blur kernel. In contrast, recent deep learning methods generally learn, from a large collection of training images, deep neural networks (DNNs) directly mapping the blurry image to the clean one or to the blur kernel, paying less attention to the physical degradation process of th…
ETH Zurich（苏黎世联邦理工学院）；University of Amsterdam（阿姆斯特丹大学）；Google Research（谷歌研究院）
We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from a single motion-blurred image. While previous approaches address the deblurring problem only in the 2D image domain, our proposed rigorous modeling of all object properties in the 3D domain enables the correct description of arbitrary object motion. This leads to significantly better image decomposition and sharper deblurring results. We model the observed appearance of a motion-blurred object as a combination of the background and a 3D object with constant translation and rotation. Our meth…
This paper proposes using a Gaussian mixture model as a prior, for solving two image inverse problems, namely image deblurring and compressive imaging. We capitalize on the fact that variable splitting algorithms, like ADMM, are able to decouple the handling of the observation operator from that of the regularizer, and plug a state-of-the-art algorithm into the pure denoising step. Furthermore, we show that, when applied to a specific type of image, a Gaussian mixture model trained from an database of images of the same type is able to outperform current state-of-the-art methods.
SDT-DCSCN for Simultaneous Super-Resolution and Deblurring of Text Images
University of Gabes（加贝斯大学）；University of Sfax（斯法克斯大学）；University of Zaragoza（萨拉戈萨大学）；University of Johannesburg（约翰内斯堡大学）；Taif University（塔伊夫大学）；King Saud University（沙特国王大学）；University of Monastir（莫纳斯提尔大学）
National Institute of Informatics, Japan（日本信息机构国家研究所）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, China（视觉技术国家工程研究中心，北京大学计算机学院，中国）；State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, China（多媒体信息处理国家重点实验室，北京大学计算机学院，中国）；National Key Laboratory of General AI, School of Intelligence Science and Technology, Peking University, China（通用人工智能国家重点实验室，北京大学智能科学与技术学院，中国）
Seoul National University（首尔大学）；Inha University（仁荷大学）
Removing camera motion blur from a single light field is a challenging task since it is highly ill-posed inverse problem. The problem becomes even worse when blur kernel varies spatially due to scene depth variation and high-order camera motion. In this paper, we propose a novel algorithm to estimate all blur model variables jointly, including latent sub-aperture image, camera motion, and scene depth from the blurred 4D light field. Exploiting multi-view nature of a light field relieves the inverse property of the optimization by utilizing strong depth cues and multi-view blur observation. The…
University of Oulu（奥卢大学）；Aalto University（阿尔托大学）；Czech Technical University in Prague（捷克技术大学布拉格分校）；Department of Computer Science（计算机科学系）；Center for Machine Perception（机器感知中心）；Faculty of Electrical Engineering（电气工程学院）；Department of Electrical Engineering and Automation（电气工程与自动化系）
The paper addresses the problem of acquiring high-quality photographs with handheld smartphone cameras in low-light imaging conditions. We propose an approach based on capturing pairs of short and long exposure images in rapid succession and fusing them into a single high-quality photograph. Unlike existing methods, we take advantage of both images simultaneously and perform a joint denoising and deblurring using a convolutional neural network. A novel approach is introduced to generate realistic short-long exposure image pairs. The method produces good images in extremely challenging conditio…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
In this paper, we propose a novel joint deblurring and multi-frame interpolation (DeMFI) framework, called DeMFI-Net, which accurately converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate based on flow-guided attentive-correlation-based feature bolstering (FAC-FB) module and recursive boosting (RB), in terms of multi-frame interpolation (MFI). The DeMFI-Net jointly performs deblurring and MFI where its baseline version performs feature-flow-based warping with FAC-FB module to obtain a sharp-interpolated frame as well to deblur two center-input frames. Moreover, its e…
Deep learning-based underwater object detection (UOD) remains a major challenge due to the degraded visibility and difficulty to obtain sufficient underwater object images captured from various perspectives for training. To address these issues, this paper presents a high-precision UOD based on joint self-supervised deblurring and improved spatial transformer network. A self-supervised deblurring subnetwork is introduced into the designed multi-task learning aided object detection architecture to force the shared feature extraction module to output clean features for detection subnetwork. Aimi…
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）
When taking photos under an environment with insufficient light, the exposure time and the sensor gain usually require to be carefully chosen to obtain images with satisfying visual quality. For example, the images with high ISO usually have inescapable noise, while the long-exposure ones may be blurry due to camera shake or object motion. Existing solutions generally suggest to seek a balance between noise and blur, and learn denoising or deblurring models under either full- or self-supervision. However, the real-world training pairs are difficult to collect, and the self-supervised methods m…
Small compression noises, despite being transparent to human eyes, can adversely affect the results of many image restoration processes, if left unaccounted for. Especially, compression noises are highly detrimental to inverse operators of high-boosting (sharpening) nature, such as deblurring and superresolution against a convolution kernel. By incorporating the non-linear DCT quantization mechanism into the formulation for image restoration, we propose a new sparsity-based convex programming approach for joint compression noise removal and image restoration. Experimental results demonstrate s…
Deep neural networks (DNNs) have shown very promising results for various image restoration (IR) tasks. However, the design of network architectures remains a major challenging for achieving further improvements. While most existing DNN-based methods solve the IR problems by directly mapping low quality images to desirable high-quality images, the observation models characterizing the image degradation processes have been largely ignored. In this paper, we first propose a denoising-based IR algorithm, whose iterative steps can be computed efficiently. Then, the iterative process is unfolded in…
Australian National University（澳大利亚国立大学）；Northwestern Polytechnical University（西北工业大学）
Stereo videos for the dynamic scenes often show unpleasant blurred effects due to the camera motion and the multiple moving objects with large depth variations. Given consecutive blurred stereo video frames, we aim to recover the latent clean images, estimate the 3D scene flow and segment the multiple moving objects. These three tasks have been previously addressed separately, which fail to exploit the internal connections among these tasks and cannot achieve optimality. In this paper, we propose to jointly solve these three tasks in a unified framework by exploiting their intrinsic connection…
Harbin Institute of Technology（哈尔滨工业大学）；City University of Hong Kong（香港城市大学）；Peng Cheng Laboratory（鹏城实验室）
Natural videos captured by consumer cameras often suffer from low framerate and motion blur due to the combination of dynamic scene complexity, lens and sensor imperfection, and less than ideal exposure setting. As a result, computational methods that jointly perform video frame interpolation and deblurring begin to emerge with the unrealistic assumption that the exposure time is known and fixed. In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame interpolation and deblurring under unknown exposure time. Toward this goal, we first adopt a varian…
Recovering sharp video sequence from a motion-blurred image is highly ill-posed due to the significant loss of motion information in the blurring process. For event-based cameras, however, fast motion can be captured as events at high time rate, raising new opportunities to exploring effective solutions. In this paper, we start from a sequential formulation of event-based motion deblurring, then show how its optimization can be unfolded with a novel end-to-end deep architecture. The proposed architecture is a convolutional recurrent neural network that integrates visual and temporal knowledge …
Facial image super-resolution (SR) is an important preprocessing for facial image analysis, face recognition, and image-based 3D face reconstruction. Recent convolutional neural network (CNN) based method has shown excellent performance by learning mapping relation using pairs of low-resolution (LR) and high-resolution (HR) facial images. However, since the HR facial image reconstruction using CNN is conventionally aimed to increase the PSNR and SSIM metrics, the reconstructed HR image might not be realistic even with high scores. An adversarial framework is proposed in this study to reconstru…
We propose a novel end-to-end learning-based approach for single image defocus deblurring. The proposed approach is equipped with a novel Iterative Filter Adaptive Network (IFAN) that is specifically designed to handle spatially-varying and large defocus blur. For adaptively handling spatially-varying blur, IFAN predicts pixel-wise deblurring filters, which are applied to defocused features of an input image to generate deblurred features. For effectively managing large blur, IFAN models deblurring filters as stacks of small-sized separable filters. Predicted separable deblurring filters are a…
Joint multi-dimensional dynamic attention and transformer for general image restoration
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education（教育部光子技术集成感知与通信重点实验室）；Guangdong Provincial Key Laboratory of Information Photonics Technology（广东省信息光子技术重点实验室）；Institute of Advanced Photonics Technology, School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院先进光子技术研究所）；School of Electronics and Communication Engineering, Sun Yat-Sen University（中山大学电子与通信工程学院）
