DeFMO: Deblurring and Shape Recovery of Fast Moving Objects
ETH Zurich（苏黎世联邦理工学院）；Microsoft Mixed Reality and AI Zurich Lab（微软苏黎世混合现实与人工智能实验室）；Google Research（谷歌研究院）；Czech Technical University in Prague（布拉格捷克理工大学）
Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Jiri Matas, Marc Pollefeys
备注：
CVPR 2021camera-ready
发表信息：
2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Objects moving at high speed appear significantly blurred when captured with cameras. The blurry appearance is especially ambiguous when the object has complex shape or texture. In such cases, classical methods, or even humans, are unable to recover the object's appearance and motion. We propose a method that, given a single image with its estimated background, outputs the object's appearance and position in a series of sub-frames as if captured by a high-speed camera (i.e. temporal super-resolution). The proposed generative model embeds an image of the blurred object into a latent space representation, disentangles the background, and renders the sharp appearance. Inspired by the image formation model, we design novel self-supervised loss function terms that boost performance and show good generalization capabilities. The proposed DeFMO method is trained on a complex synthetic dataset, yet it performs well on real-world data from several datasets. DeFMO outperforms the state of the art and generates high-quality temporal super-resolution frames.
Recent approaches employ deep learning-based solutions for the recovery of a sharp image from its blurry observation. This paper introduces adversarial attacks against deep learning-based image deblurring methods and evaluates the robustness of these neural networks to untargeted and targeted attacks. We demonstrate that imperceptible distortion can significantly degrade the performance of state-of-the-art deblurring networks, even producing drastically different content in the output, indicating the strong need to include adversarially robust training not only in classification but also for i…
University of Electronic Science and Technology of China（电子科技大学）；Cornell University（康奈尔大学）
The wavelet frame systems have been playing an active role in image restoration and many other image processing fields over the past decades, owing to the good capability of sparsely approximating piece-wise smooth functions such as images. In this paper, we propose a novel wavelet frame based sparse recovery model called \textit{Support Driven Sparse Regularization} (SDSR) for image deblurring, where the partial support information of frame coefficients is attained via a self-learning strategy and exploited via the proposed truncated $\ell_0$ regularization. Moreover, the state-of-the-art ima…
Third Military Medical University（第三军医大学）；National University of Defense Technology（国防科技大学）
The recovery of images from the observations that are degraded by a linear operator and further corrupted by Poisson noise is an important task in modern imaging applications such as astronomical and biomedical ones. Gradient-based regularizers involve the popular total variation semi-norm have become standard techniques for Poisson image restoration due to its edge-preserving ability. Various efficient algorithms have been developed for solving the corresponding minimization problem with non-smooth regularization terms. In this paper, motivated by the idea of the alternating direction minimiz…
National Laboratory of Radar Signal Processing, Xidian University（西安电子科技大学雷达信号处理国家重点实验室）；Science and Technology on Multispectral Information Processing Laboratory, School of Automation, Huazhong University of Science and Technology（华中科技大学自动化学院多谱信息处理技术国家级重点实验室）
Hyperspectral imaging, providing abundant spatial and spectral information simultaneously, has attracted a lot of interest in recent years. Unfortunately, due to the hardware limitations, the hyperspectral image (HSI) is vulnerable to various degradations, such noises (random noise, HSI denoising), blurs (Gaussian and uniform blur, HSI deblurring), and down-sampled (both spectral and spatial downsample, HSI super-resolution). Previous HSI restoration methods are designed for one specific task only. Besides, most of them start from the 1-D vector or 2-D matrix models and cannot fully exploit th…
UHD Image Deblurring via Autoregressive Flow with Ill-conditioned Constraints
Shandong Normal University（山东师范大学）；Qilu University of Technology（齐鲁工业大学）；Nanjing University of Science and Technology（南京理工大学）；National University of Defense Technology（国防科学技术大学）；Shandong University（山东大学）
University of Sydney（悉尼大学）；Birkbeck College（伯贝克学院）
Motion blur in dynamic scenes is an important yet challenging research topic. Recently, deep learning methods have achieved impressive performance for dynamic scene deblurring. However, the motion information contained in a blurry image has yet to be fully explored and accurately formulated because: (i) the ground truth of dynamic motion is difficult to obtain; (ii) the temporal ordering is destroyed during the exposure; and (iii) the motion estimation from a blurry image is highly ill-posed. By revisiting the principle of camera exposure, motion blur can be described by the relative motions o…
Australian National University（澳大利亚国立大学）；Northwestern Polytechnical University（西北工业大学）；Australia Centre for Robotic Vision（澳大利亚机器人视觉中心）
Camera shake during exposure is a major problem in hand-held photography, as it causes image blur that destroys details in the captured images.~In the real world, such blur is mainly caused by both the camera motion and the complex scene structure.~While considerable existing approaches have been proposed based on various assumptions regarding the scene structure or the camera motion, few existing methods could handle the real 6 DoF camera motion.~In this paper, we propose to jointly estimate the 6 DoF camera motion and remove the non-uniform blur caused by camera motion by exploiting their un…
Rice University（莱斯大学）；University of California（加利福尼亚大学）
Various algorithms have been proposed for dictionary learning. Among those for image processing, many use image patches to form dictionaries. This paper focuses on whole-image recovery from corrupted linear measurements. We address the open issue of representing an image by overlapping patches: the overlapping leads to an excessive number of dictionary coefficients to determine. With very few exceptions, this issue has limited the applications of image-patch methods to the local kind of tasks such as denoising, inpainting, cartoon-texture decomposition, super-resolution, and image deblurring, …
Depth Estimation and Image Restoration by Deep Learning from Defocused Images
CEOSpaceTech；University POLITEHNICA of Bucharest（布加勒斯特理工大学）；Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS)（智能技术卓越研究中心（CiTIUS））；University of Santiago de Compostela（圣地亚哥德孔波斯特拉大学）
Compressive sensing promises to enable bandwidth-efficient on-board compression of astronomical data by lifting the encoding complexity from the source to the receiver. The signal is recovered off-line, exploiting GPUs parallel computation capabilities to speedup the reconstruction process. However, inherent GPU hardware constraints limit the size of the recoverable signal and the speedup practically achievable. In this work, we design parallel algorithms that exploit the properties of circulant matrices for efficient GPU-accelerated sparse signals recovery. Our approach reduces the memory req…
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of British Columbia（不列颠哥伦比亚大学）
Much recent attention has been devoted to gradient descent algorithms where the steepest descent step size is replaced by a similar one from a previous iteration or gets updated only once every second step, thus forming a {\em faster gradient descent method}. For unconstrained convex quadratic optimization these methods can converge much faster than steepest descent. But the context of interest here is application to certain ill-posed inverse problems, where the steepest descent method is known to have a smoothing, regularizing effect, and where a strict optimization solution is not necessary.…
Washington University in St. Louis（圣路易斯华盛顿大学）；Department of Electrical and Systems Engineering（电气与系统工程系）；Department of Computer Science and Engineering（计算机科学与工程系）
Regularization by denoising (RED) is a widely-used framework for solving inverse problems by leveraging image denoisers as image priors. Recent work has reported the state-of-the-art performance of RED in a number of imaging applications using pre-trained deep neural nets as denoisers. Despite the recent progress, the stable convergence of RED algorithms remains an open problem. The existing RED theory only guarantees stability for convex data-fidelity terms and nonexpansive denoisers. This work addresses this issue by developing a new monotone RED (MRED) algorithm, whose convergence does not …
Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Image quality is the basis of image communication and understanding tasks. Due to the blur and noise effects caused by imaging, transmission and other processes, the image quality is degraded. Blind image restoration is widely used to improve image quality, where the main goal is to faithfully estimate the blur kernel and the latent sharp image. In this study, based on experimental observation and research, an adaptively sparse regularized minimization method is originally proposed. The high-order gradients combine with low-order ones to form a hybrid regularization term, and an adaptive opera…
We present a method that takes as input a single dual-pixel image, and simultaneously estimates the image's defocus map -- the amount of defocus blur at each pixel -- and recovers an all-in-focus image. Our method is inspired from recent works that leverage the dual-pixel sensors available in many consumer cameras to assist with autofocus, and use them for recovery of defocus maps or all-in-focus images. These prior works have solved the two recovery problems independently of each other, and often require large labeled datasets for supervised training. By contrast, we show that it is beneficia…
Since the matrix formed by nonlocal similar patches in a natural image is of low rank, the nuclear norm minimization (NNM) has been widely used in various image processing studies. Nonetheless, nuclear norm based convex surrogate of the rank function usually over-shrinks the rank components and makes different components equally, and thus may produce a result far from the optimum. To alleviate the above-mentioned limitations of the nuclear norm, in this paper we propose a new method for image restoration via the non-convex weighted Lp nuclear norm minimization (NCW-NNM), which is able to more …
Technion Israel Institute of Technology（以色列理工学院）；Google Research（谷歌研究院）
Inverse problems in image processing are typically cast as optimization tasks, consisting of data-fidelity and stabilizing regularization terms. A recent regularization strategy of great interest utilizes the power of denoising engines. Two such methods are the Plug-and-Play Prior (PnP) and Regularization by Denoising (RED). While both have shown state-of-the-art results in various recovery tasks, their theoretical justification is incomplete. In this paper, we aim to bridge between RED and PnP, enriching the understanding of both frameworks. Towards that end, we reformulate RED as a convex op…
The vast majority of image recovery tasks are ill-posed problems. As such, methods that are based on optimization use cost functions that consist of both fidelity and prior (regularization) terms. A recent line of works imposes the prior by the Regularization by Denoising (RED) approach, which exploits the good performance of existing image denoising engines. Yet, the relation of RED to explicit prior terms is still not well understood, as previous work requires too strong assumptions on the denoisers. In this paper, we make two contributions. First, we show that the RED gradient can be seen a…
Inception Institute of AI（Inception人工智能研究所）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch t…
School of Informatics（信息学院）；Xiamen University（厦门大学）
Low-quality face image restoration is a popular research direction in today's computer vision field. It can be used as a pre-work for tasks such as face detection and face recognition. At present, there is a lot of work to solve the problem of low-quality faces under various environmental conditions. This paper mainly focuses on the restoration of motion-blurred faces. In increasingly abundant mobile scenes, the fast recovery of motion-blurred faces can bring highly effective speed improvements in tasks such as face matching. In order to achieve this goal, a deblurring method for motion-blurre…
Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）
Traditional patch-based sparse representation modeling of natural images usually suffer from two problems. First, it has to solve a large-scale optimization problem with high computational complexity in dictionary learning. Second, each patch is considered independently in dictionary learning and sparse coding, which ignores the relationship among patches, resulting in inaccurate sparse coding coefficients. In this paper, instead of using patch as the basic unit of sparse representation, we exploit the concept of group as the basic unit of sparse representation, which is composed of nonlocal p…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Existing works on motion deblurring either ignore the effects of depth-dependent blur or work with the assumption of a multi-layered scene wherein each layer is modeled in the form of fronto-parallel plane. In this work, we consider the case of 3D scenes with piecewise planar structure i.e., a scene that can be modeled as a combination of multiple planes with arbitrary orientations. We first propose an approach for estimation of normal of a planar scene from a single motion blurred observation. We then develop an algorithm for automatic recovery of number of planes, the parameters correspondin…
Correlation Matching Transformation Transformers for UHD Image Restoration
The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）；National University of Singapore（新加坡国立大学）；Hebei University of Technology（河北工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Ultra-High-Definition Image Restoration: New Benchmarks and A Dual Interaction Prior-Driven Solution
Dalian University of Technology（大连理工大学）；Centre for Advances in Reliability and Safety（可靠性与安全先进中心）；Nanjing University of Science and Technology（南京理工大学）；Yale University（耶鲁大学）；Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province（辽宁省计算数学与数据智能重点实验室）
We present a method to extract a video sequence from a single motion-blurred image. Motion-blurred images are the result of an averaging process, where instant frames are accumulated over time during the exposure of the sensor. Unfortunately, reversing this process is nontrivial. Firstly, averaging destroys the temporal ordering of the frames. Secondly, the recovery of a single frame is a blind deconvolution task, which is highly ill-posed. We present a deep learning scheme that gradually reconstructs a temporal ordering by sequentially extracting pairs of frames. Our main contribution is to i…
DocPure: Prompt-Free Unified Document Restoration via Degradation-Aware Structure-Guided Wavelet Modulation
College of Computer Science and Artificial Intelligence, Wenzhou University（温州大学计算机与人工智能学院）；AI Analytics Team, Nasdaq（纳斯达克人工智能分析团队）；vivo Mobile Communication Co., Ltd（维沃移动通信有限公司）；College of Engineering and Computer Science, The Australian National University（澳大利亚国立大学工程与计算机科学学院）
