Image deblurring is an ill-posed problem with multiple plausible solutions for a given input image. However, most existing methods produce a deterministic estimate of the clean image and are trained to minimize pixel-level distortion. These metrics are known to be poorly correlated with human perception, and often lead to unrealistic reconstructions. We present an alternative framework for blind deblurring based on conditional diffusion models. Unlike existing techniques, we train a stochastic sampler that refines the output of a deterministic predictor and is capable of producing a diverse set of plausible reconstructions for a given input. This leads to a significant improvement in perceptual quality over existing state-of-the-art methods across multiple standard benchmarks. Our predict-and-refine approach also enables much more efficient sampling compared to typical diffusion models. Combined with a carefully tuned network architecture and inference procedure, our method is competitive in terms of distortion metrics such as PSNR. These results show clear benefits of our diffusion-based method for deblurring and challenge the widely used strategy of producing a single, deterministic reconstruction.
DWDN: Deep Wiener Deconvolution Network for Non-Blind Image Deblurring
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Department of Computer Science, TU Darmstadt（达姆施塔特工业大学计算机科学系）
UHD Image Deblurring via Autoregressive Flow with Ill-conditioned Constraints
Shandong Normal University（山东师范大学）；Qilu University of Technology（齐鲁工业大学）；Nanjing University of Science and Technology（南京理工大学）；National University of Defense Technology（国防科学技术大学）；Shandong University（山东大学）
Peking University（北京大学）；Cooperative Media-net Innovation Center（协同媒体网创新中心）；Beida (Binhai) Information Research（北大（滨海）信息研究院）；Harbin Institute of Technology（哈尔滨工业大学）
Blind image deblurring is a challenging problem in computer vision, which aims to restore both the blur kernel and the latent sharp image from only a blurry observation. Inspired by the prevalent self-example prior in image super-resolution, in this paper, we observe that a coarse enough image down-sampled from a blurry observation is approximately a low-resolution version of the latent sharp image. We prove this phenomenon theoretically and define the coarse enough image as a latent structure prior of the unknown sharp image. Starting from this prior, we propose to restore sharp images from t…
Laboratoire d’Informatique de l’École Normale Supérieure（巴黎高等师范学院计算机实验室）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；University of California, Berkeley（加州大学伯克利分校）
This paper proposes a novel approach to image deblurring and digital zooming using sparse local models of image appearance. These models, where small image patches are represented as linear combinations of a few elements drawn from some large set (dictionary) of candidates, have proven well adapted to several image restoration tasks. A key to their success has been to learn dictionaries adapted to the reconstruction of small image patches. In contrast, recent works have proposed instead to learn dictionaries which are not only adapted to data reconstruction, but also tuned for a specific task.…
Image restoration has been an extensively researched topic in numerous fields. With the advent of deep learning, a lot of the current algorithms were replaced by algorithms that are more flexible and robust. Deep networks have demonstrated impressive performance in a variety of tasks like blind denoising, image enhancement, deblurring, super-resolution, inpainting, among others. Most of these learning-based algorithms use a large amount of clean data during the training process. However, in certain applications in medical image processing, one may not have access to a large amount of clean dat…
U-DAVI: Uncertainty-Aware Diffusion-Prior-Based Amortized Variational Inference for Image Reconstruction
Computing and Mathematical Sciences, California Institute of Technology（计算与数学科学系，加州理工学院）；Mathematical Sciences, California Institute of Technology（数学科学系，加州理工学院）
We present SR3, an approach to image Super-Resolution via Repeated Refinement. SR3 adapts denoising diffusion probabilistic models to conditional image generation and performs super-resolution through a stochastic denoising process. Inference starts with pure Gaussian noise and iteratively refines the noisy output using a U-Net model trained on denoising at various noise levels. SR3 exhibits strong performance on super-resolution tasks at different magnification factors, on faces and natural images. We conduct human evaluation on a standard 8X face super-resolution task on CelebA-HQ, comparing…
University of Edinburgh（爱丁堡大学）；Noiseless Imaging Ltd（Noiseless成像有限公司）；University of Bath（巴斯大学）
In this work we investigate the practicality of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the deterministic gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-in…
Efficient Terrain Stochastic Differential Efficient Terrain Stochastic Differential Equations for Multipurpose Digital Elevation Model Restoration
School of Aeronautics and Astronautics, Shanghai Jiao Tong University（上海交通大学航空航天学院）；Land Satellite Remote Sensing Application Center（陆地卫星遥感应用中心）；Ministry of Natural Resources of the People’s Republic of China（中华人民共和国自然资源部）；China-ASEAN Satellite Remote Sensing Applications（中国-东盟卫星遥感应用（中心））
Most image deblurring methods assume an over-simplistic image formation model and as a result are sensitive to more realistic image degradations. We propose a novel variational framework, that explicitly handles pixel saturation, noise, quantization, as well as non-linear camera response function due to e.g., gamma correction. We show that accurately modeling a more realistic image acquisition pipeline leads to significant improvements, both in terms of image quality and PSNR. Furthermore, we show that incorporating the non-linear response in both the data and the regularization terms of the p…
University of Fukui（福井大学）；Deakin University（迪肯大学）；Nanyang Technological University（南洋理工大学）
Complex blur such as the mixup of space-variant and space-invariant blur, which is hard to model mathematically, widely exists in real images. In this paper, we propose a novel image deblurring method that does not need to estimate blur kernels. We utilize a pair of images that can be easily acquired in low-light situations: (1) a blurred image taken with low shutter speed and low ISO noise; and (2) a noisy image captured with high shutter speed and high ISO noise. Slicing the blurred image into patches, we extend the Gaussian mixture model (GMM) to model the underlying intensity distribution …
Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sa…
Boosting Image Restoration via Priors from Pre-trained Models
Zhejiang Lab（之江实验室）；CUHK（香港中文大学）；RealityEdge（睿迪智芯）；University of Macau（澳门大学）；Institute of Collaborative Innovation（协同创新研究院）；Texas A&M University（德克萨斯农工大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）
In semantic segmentation tasks, input images can often have more than one plausible interpretation, thus allowing for multiple valid labels. To capture such ambiguities, recent work has explored the use of probabilistic networks that can learn a distribution over predictions. However, these do not necessarily represent the empirical distribution accurately. In this work, we present a strategy for learning a calibrated predictive distribution over semantic maps, where the probability associated with each prediction reflects its ground truth correctness likelihood. To this end, we propose a nove…
Nanjing University of Science and Technology（南京理工大学）；Tianjin AiForward Science and Technology Co., Ltd.（天津爱前进科技有限公司）
Stochastic human motion prediction aims to forecast multiple plausible future motions given a single pose sequence from the past. Most previous works focus on designing elaborate losses to improve the accuracy, while the diversity is typically characterized by randomly sampling a set of latent variables from the latent prior, which is then decoded into possible motions. This joint training of sampling and decoding, however, suffers from posterior collapse as the learned latent variables tend to be ignored by a strong decoder, leading to limited diversity. Alternatively, inspired by the diffusi…
Integrated Forward-Inverse Network for Lensless Image Reconstruction
Department of Mechanical Engineering, Seoul National University（韩国首尔国立大学机械工程系）；School of Mechanical and Aerospace Engineering/SNU-IAMD, Seoul National University（韩国首尔国立大学机械与航空航天工程学院/SNU-IAMD）；Department of Electrical and Electronic Engineering, Yonsei University（韩国延世大学电气与电子工程系）；Department of Biomedical Engineering, University of Michigan（美国密歇根大学安娜堡分校生物医学工程系）
Google DeepMind（谷歌DeepMind）；Applied AI Lab, University of Oxford（牛津大学应用人工智能实验室）
Advances in unsupervised learning of object-representations have culminated in the development of a broad range of methods for unsupervised object segmentation and interpretable object-centric scene generation. These methods, however, are limited to simulated and real-world datasets with limited visual complexity. Moreover, object representations are often inferred using RNNs which do not scale well to large images or iterative refinement which avoids imposing an unnatural ordering on objects in an image but requires the a priori initialisation of a fixed number of object representations. In c…
Diff-PCR: Diffusion-Based Correspondence Searching in Doubly Stochastic Matrix Space for Point Cloud Registration
Jinling Institute of Technology（金陵科技学院）；PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院，教育部高维信息智能感知与系统重点实验室，江苏省社会安全图像与视频理解重点实验室，PCA实验室）
