Existing image restoration methods mostly leverage the posterior distribution of natural images. However, they often assume known degradation and also require supervised training, which restricts their adaptation to complex real applications. In this work, we propose the Generative Diffusion Prior (GDP) to effectively model the posterior distributions in an unsupervised sampling manner. GDP utilizes a pre-train denoising diffusion generative model (DDPM) for solving linear inverse, non-linear, or blind problems. Specifically, GDP systematically explores a protocol of conditional guidance, which is verified more practical than the commonly used guidance way. Furthermore, GDP is strength at optimizing the parameters of degradation model during the denoising process, achieving blind image restoration. Besides, we devise hierarchical guidance and patch-based methods, enabling the GDP to generate images of arbitrary resolutions. Experimentally, we demonstrate GDP's versatility on several image datasets for linear problems, such as super-resolution, deblurring, inpainting, and colorization, as well as non-linear and blind issues, such as low-light enhancement and HDR image recovery. GDP outperforms the current leading unsupervised methods on the diverse benchmarks in reconstruction quality and perceptual quality. Moreover, GDP also generalizes well for natural images or synthesized images with arbitrary sizes from various tasks out of the distribution of the ImageNet training set.
V-Bridge: Bridging Video Generative Priors to Versatile Few-shot Image Restoration
The Hong Kong University of Science and Technology（香港科学与技术大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）；SenseTime Research（商汤科技研究院）；Tencent（腾讯）；University of California, Merced（加州大学默塞德分校）
We present an algorithm to directly solve numerous image restoration problems (e.g., image deblurring, image dehazing, image deraining, etc.). These problems are highly ill-posed, and the common assumptions for existing methods are usually based on heuristic image priors. In this paper, we find that these problems can be solved by generative models with adversarial learning. However, the basic formulation of generative adversarial networks (GANs) does not generate realistic images, and some structures of the estimated images are usually not preserved well. Motivated by an interesting observati…
University of Notre Dame（圣母大学）；University of Ljubljana（卢布尔雅那大学）
Advances in image restoration and enhancement techniques have led to discussion about how such algorithmscan be applied as a pre-processing step to improve automatic visual recognition. In principle, techniques like deblurring and super-resolution should yield improvements by de-emphasizing noise and increasing signal in an input image. But the historically divergent goals of the computational photography and visual recognition communities have created a significant need for more work in this direction. To facilitate new research, we introduce a new benchmark dataset called UG^2, which contain…
Neural Discrimination-Prompted Transformers for Efficient UHD Image Restoration and Enhancement
Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）；University of California, San Francisco（加州大学旧金山分校）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lingnan University（岭南大学）；Hong Kong Metropolitan University（香港都会大学）
Adverse weather conditions such as haze, rain, and snow often impair the quality of captured images, causing detection networks trained on normal images to generalize poorly in these scenarios. In this paper, we raise an intriguing question - if the combination of image restoration and object detection, can boost the performance of cutting-edge detectors in adverse weather conditions. To answer it, we propose an effective yet unified detection paradigm that bridges these two subtasks together via dynamic enhancement learning to discern objects in adverse weather conditions, called TogetherNet.…
A Generative Data Framework with Authentic Supervision for Underwater Image Restoration and Enhancement
Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）；Department of Automation, Shanghai Jiao Tong University（自动化系，上海交通大学）；School of Artificial Intelligence, Optics and ElectroNics (iOPEN), Northwestern Polytechnical University（人工智能、光学与电子学学院（iOPEN），西北工业大学）；Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（西安光学精密机械研究所，中国科学院）
IR-Flow: Bridging Discriminative and Generative Image Restoration via Rectified Flow
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, School of Information Science and Technology, University of Science and Technology of China（脑启发智能感知与认知联合实验室，信息科学与技术学院，中国科学技术大学）
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；The Chinese University of Hong Kong（香港中文大学）
How far have we gone in Generative Image Restoration? A study on its capability, limitations and evaluation practices
Fudan University（复旦大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of the Chinese Academy of Sciences（中国科学院大学）；Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Tencent（腾讯）；The Chinese University of Hong Kong at Shenzhen（香港中文大学（深圳））
Restoring the clean background from the superimposed images containing a noisy layer is the common crux of a classical category of tasks on image restoration such as image reflection removal, image deraining and image dehazing. These tasks are typically formulated and tackled individually due to the diverse and complicated appearance patterns of noise layers within the image. In this work we present the Deep-Masking Generative Network (DMGN), which is a unified framework for background restoration from the superimposed images and is able to cope with different types of noise. Our proposed DMGN…
Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey
University of Science and Technology of China（科学技术大学）；National University of Singapore（国立新加坡大学）；Eastern Institute for Advanced Study（东部高级研究机构）；Microsoft Research Asia（微软亚洲研究院）
End-to-end generative methods are considered a more promising solution for image restoration in physics-based vision compared with the traditional deconstructive methods based on handcrafted composition models. However, existing generative methods still have plenty of room for improvement in quantitative performance. More crucially, these methods are considered black boxes due to weak interpretability and there is rarely a theory trying to explain their mechanism and learning process. In this study, we try to re-interpret these generative methods for image restoration tasks using information t…
Inception Institute of Artificial Intelligence（inception人工智能研究院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）
With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, sem…
Academy for Engineering & Technology, Fudan University（复旦大学工程与应用技术研究院）；College of Information Technology, Shanghai Ocean University（上海海洋大学信息学院）；University of Calabria（卡拉布里亚大学）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Napier Edinburgh University（爱丁堡龙比亚大学）
Underwater images play a key role in ocean exploration, but often suffer from severe quality degradation due to light absorption and scattering in water medium. Although major breakthroughs have been made recently in the general area of image enhancement and restoration, the applicability of new methods for improving the quality of underwater images has not specifically been captured. In this paper, we review the image enhancement and restoration methods that tackle typical underwater image impairments, including some extreme degradations and distortions. Firstly, we introduce the key causes o…
CRNet: A Detail-Preserving Network for Unified Image Restoration and Enhancement Task
Northwestern Polytechnical University（西北工业大学）；Xi’an University of Architecture and Technology（西安建筑科技大学）；Xi’an Institute of Optics and Precision Mechanics of CAS（中国科学院西安光学精密机械研究所）
Bracketing Image Restoration and Enhancement with High-Low Frequency Decomposition
Xi’an University of Architecture and Technology（西安建筑科技大学）；Northwestern Polytechnical University（西北工业大学）；University of Macau（澳门大学）；Xi’an Institute of Optics and Precision Mechanics of CAS（中国科学院西安光学精密机械研究所）
National Tsing Hua University（国立清华大学）；Chalmers University of Technology（查尔姆斯理工大学）；Universiti Malaya（马来亚大学）；Microsoft AI R & D Center（微软人工智能研发中心）
Deep learning-based methods have made impressive progress in enhancing extremely low-light images - the image quality of the reconstructed images has generally improved. However, we found out that most of these methods could not sufficiently recover the image details, for instance, the texts in the scene. In this paper, a novel image enhancement framework is proposed to precisely restore the scene texts, as well as the overall quality of the image simultaneously under extremely low-light images conditions. Mainly, we employed a self-regularised attention map, an edge map, and a novel text dete…
Rutgers University（罗格斯大学）；University of Patras（帕特雷大学）
When images are statistically described by a generative model we can use this information to develop optimum techniques for various image restoration problems as inpainting, super-resolution, image coloring, generative model inversion, etc. With the help of the generative model it is possible to formulate, in a natural way, these restoration problems as Statistical estimation problems. Our approach, by combining maximum a-posteriori probability with maximum likelihood estimation, is capable of restoring images that are distorted by transformations even when the latter contain unknown parameter…
Gated Multi-Resolution Transfer Network for Burst Restoration and Enhancement
Indian Institute of Technology Ropar（印度理工学院鲁尔基分校）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of AI（起源人工智能研究院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
This work presents an unsupervised deep learning scheme that exploiting high-dimensional assisted score-based generative model for color image restoration tasks. Considering that the sample number and internal dimension in score-based generative model have key influence on estimating the gradients of data distribution, two different high-dimensional ways are proposed: The channel-copy transformation increases the sample number and the pixel-scale transformation decreases feasible space dimension. Subsequently, a set of high-dimensional tensors represented by these transformations are used to t…
Boosting Image Restoration via Priors from Pre-trained Models
Zhejiang Lab（之江实验室）；CUHK（香港中文大学）；RealityEdge（睿迪智芯）；University of Macau（澳门大学）；Institute of Collaborative Innovation（协同创新研究院）；Texas A&M University（德克萨斯农工大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of AI（英赛普申人工智能研究院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
Modern handheld devices can acquire burst image sequence in a quick succession. However, the individual acquired frames suffer from multiple degradations and are misaligned due to camera shake and object motions. The goal of Burst Image Restoration is to effectively combine complimentary cues across multiple burst frames to generate high-quality outputs. Towards this goal, we develop a novel approach by solely focusing on the effective information exchange between burst frames, such that the degradations get filtered out while the actual scene details are preserved and enhanced. Our central id…
Shanghai Jiao Tong University（上海交通大学）；Texas A&M University（得克萨斯农工大学）；University of Macau（澳门大学）
Liquify is a common technique for image editing, which can be used for image distortion. Due to the uncertainty in the distortion variation, restoring distorted images caused by liquify filter is a challenging task. To edit images in an efficient way, distorted images are expected to be restored automatically. This paper aims at the distorted image restoration, which is characterized by seeking the appropriate warping and completion of a distorted image. Existing methods focus on the hardware assistance or the geometric principle to solve the specific regular deformation caused by natural phen…
Southern University of Science and Technology（南方科技大学）；A*STAR（新加坡科技研究局）；New York University（纽约大学）；Chinese Academy of Sciences（中国科学院）；Shenzhen People’s Hospital（深圳市人民医院）
Fundus photography is a routine examination in clinics to diagnose and monitor ocular diseases. However, for cataract patients, the fundus image always suffers quality degradation caused by the clouding lens. The degradation prevents reliable diagnosis by ophthalmologists or computer-aided systems. To improve the certainty in clinical diagnosis, restoration algorithms have been proposed to enhance the quality of fundus images. Unfortunately, challenges remain in the deployment of these algorithms, such as collecting sufficient training data and preserving retinal structures. In this paper, to …
Burstormer: Burst Image Restoration and Enhancement Transformer
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of AI（Inception人工智能研究院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；The University of Hong Kong（香港大学）
Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig.1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, p…
Deep Learning-Driven Ultra-High-Definition Image Restoration: A Survey
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；Centre for Advances in Reliability and Safety, Hong Kong, China（香港可靠性与安全中心）；Hong Kong Polytechnic University（香港理工大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Shandong University of Traditional Chinese Medicine（山东中医药大学）；Nankai University（南开大学）；ShanghaiTech University（上海科技大学）；Taishan University（泰山学院）；United Imaging Co.（联影公司）
Real-world face super-resolution (SR) is a highly ill-posed image restoration task. The fully-cycled Cycle-GAN architecture is widely employed to achieve promising performance on face SR, but prone to produce artifacts upon challenging cases in real-world scenarios, since joint participation in the same degradation branch will impact final performance due to huge domain gap between real-world and synthetic LR ones obtained by generators. To better exploit the powerful generative capability of GAN for real-world face SR, in this paper, we establish two independent degradation branches in the fo…
Unified Restoration-Perception Learning: Maritime Infrared-Visible Image Fusion and Segmentation
Laboratory of Underwater Acoustic Communication and Marine Information Technology, Ministry of Education, Xiamen University（厦门大学水声通信与海洋信息技术教育部重点实验室）；PAMI Research Group, Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系PAMI研究组）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）
