How far have we gone in Generative Image Restoration? A study on its capability, limitations and evaluation practices
Fudan University（复旦大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of the Chinese Academy of Sciences（中国科学院大学）；Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Generative Image Restoration (GIR) has achieved impressive perceptual realism, but how far have its practical capabilities truly advanced compared with previous methods? To answer this, we present a large-scale study grounded in a new multi-dimensional evaluation pipeline that assesses models on detail, sharpness, semantic correctness, and overall quality. Our analysis covers diverse architectures, including diffusion-based, GAN-based, PSNR-oriented, and general-purpose generation models, revealing critical performance disparities. Furthermore, our analysis uncovers a key evolution in failure modes that signifies a paradigm shift for the perception-oriented low-level vision field. The central challenge is evolving from the previous problem of detail scarcity (under-generation) to the new frontier of detail quality and semantic control (preventing over-generation). We also leverage our benchmark to train a new IQA model that better aligns with human perceptual judgments. Ultimately, this work provides a systematic study of modern generative image restoration models, offering crucial insights that redefine our understanding of their true state and chart a course for future development.
University of Science and Technology of China（中国科学技术大学）；University of Macau（澳门大学）；University of Chinese Academy of Sciences（中国科学院大学）
In this paper, we present Uformer, an effective and efficient Transformer-based architecture for image restoration, in which we build a hierarchical encoder-decoder network using the Transformer block. In Uformer, there are two core designs. First, we introduce a novel locally-enhanced window (LeWin) Transformer block, which performs nonoverlapping window-based self-attention instead of global self-attention. It significantly reduces the computational complexity on high resolution feature map while capturing local context. Second, we propose a learnable multi-scale restoration modulator in the…
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；The Chinese University of Hong Kong（香港中文大学）
Nanjing University（南京大学）；University of Adelaide（阿德莱德大学）
Image restoration, including image denoising, super resolution, inpainting, and so on, is a well-studied problem in computer vision and image processing, as well as a test bed for low-level image modeling algorithms. In this work, we propose a very deep fully convolutional auto-encoder network for image restoration, which is a encoding-decoding framework with symmetric convolutional-deconvolutional layers. In other words, the network is composed of multiple layers of convolution and de-convolution operators, learning end-to-end mappings from corrupted images to the original ones. The convoluti…
V-Bridge: Bridging Video Generative Priors to Versatile Few-shot Image Restoration
The Hong Kong University of Science and Technology（香港科学与技术大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The Hong Kong Polytechnic University（香港理工大学）
In supervised image restoration tasks, one key issue is how to obtain the aligned high-quality (HQ) and low-quality (LQ) training image pairs. Unfortunately, such HQ-LQ training pairs are hard to capture in practice, and hard to synthesize due to the complex unknown degradation in the wild. While several sophisticated degradation models have been manually designed to synthesize LQ images from their HQ counterparts, the distribution gap between the synthesized and real-world LQ images remains large. We propose a new approach to synthesizing realistic image restoration training pairs using the e…
Bridging Degradation Discrimination and Generation for Universal Image Restoration
Institute of Medical Technology, Peking University（北京大学医学技术学院）；Biomedical Engineering Department, College of Future Technology, Peking University（北京大学未来技术学院生物医学工程系）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）；China Mobile Research Institute（中国移动研究院）
A Regularization-Guided Equivariant Approach for Image Restoration
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, China（西安交通大学数学与统计学学院）；Macau University of Science and Technology, Taipa, Macao（澳门科学技术大学）；Pengcheng Laboratory, Shenzhen, China（鹏城实验室）
University of Electronic Science and Technology of China（电子科技大学）
The wavelet frame systems have been widely investigated and applied for image restoration and many other image processing problems over the past decades, attributing to their good capability of sparsely approximating piece-wise smooth functions such as images. Most wavelet frame based models exploit the $l_1$ norm of frame coefficients for a sparsity constraint in the past. The authors in \cite{ZhangY2013, Dong2013} proposed an $l_0$ minimization model, where the $l_0$ norm of wavelet frame coefficients is penalized instead, and have demonstrated that significant improvements can be achieved c…
Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；Nankai University（南开大学）
Image and video restoration has achieved a remarkable leap with the advent of deep learning. The success of deep learning paradigm lies in three key components: data, model, and loss. Currently, many efforts have been devoted to the first two while seldom study focuses on loss function. With the question ``are the de facto optimization functions e.g., $L_1$, $L_2$, and perceptual losses optimal?'', we explore the potential of loss and raise our belief ``learned loss function empowers the learning capability of neural networks for image and video restoration''. Concretely, we stand on the shoul…
National University of Defense Technology（国防科技大学）；National Innovation Institute of Defense Technology（国防科技创新研究院）
The great success of Deep Neural Networks (DNNs) has inspired the algorithmic development of DNN-based Fixed-Point (DNN-FP) for computer vision tasks. DNN-FP methods, trained by Back-Propagation Through Time or computing the inaccurate inversion of the Jacobian, suffer from inferior representation ability. Motivated by the representation power of the Transformer, we propose a framework to unroll the FP and approximate each unrolled process via Transformer blocks, called FPformer. To reduce the high consumption of memory and computation, we come up with FPRformer by sharing parameters between t…
National Laboratory of Radar Signal Processing, Xidian University（西安电子科技大学雷达信号处理国家重点实验室）；Science and Technology on Multispectral Information Processing Laboratory, School of Automation, Huazhong University of Science and Technology（华中科技大学自动化学院多谱信息处理技术国家级重点实验室）
Hyperspectral imaging, providing abundant spatial and spectral information simultaneously, has attracted a lot of interest in recent years. Unfortunately, due to the hardware limitations, the hyperspectral image (HSI) is vulnerable to various degradations, such noises (random noise, HSI denoising), blurs (Gaussian and uniform blur, HSI deblurring), and down-sampled (both spectral and spatial downsample, HSI super-resolution). Previous HSI restoration methods are designed for one specific task only. Besides, most of them start from the 1-D vector or 2-D matrix models and cannot fully exploit th…
Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；Xi’an Jiaotong University（西安交通大学）；Nankai University（南开大学）；SenseTime Research（商汤科技研究院）
In this paper, orthogonal to the existing data and model studies, we instead resort our efforts to investigate the potential of loss function in a new perspective and present our belief ``Random Weights Networks can Be Acted as Loss Prior Constraint for Image Restoration''. Inspired by Functional theory, we provide several alternative solutions to implement our belief in the strict mathematical manifolds including Taylor's Unfolding Network, Invertible Neural Network, Central Difference Convolution and Zero-order Filtering as ``random weights network prototype'' with respect of the following f…
Civil Aviation Flight University of China（中国民用航空飞行学院）；University of Electronic Science and Technology of China（电子科技大学）；Institute of North Electronic Equipment（北方电子设备研究所）；Pensees Ptd Ltd（澎思科技有限公司）；Baidu Research（百度研究院）；Tencent AI Lab（腾讯人工智能实验室）；National University of Singapore（新加坡国立大学）
In real-world scenarios, many factors may harm face recognition performance, e.g., large pose, bad illumination,low resolution, blur and noise. To address these challenges, previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition. However, most of these methods are stage-wise, which is sub-optimal and deviates from the reality. In this paper, we address all these challenges jointly for unconstrained face recognition. We propose an Multi-Degradation Face Restoration (MDFR) model to restore frontalized high-quality faces from the given l…
LoopMoEVR: Loop-Based Degradation-Aware Mixture-of-Experts for Unified UHD Video Restoration
Shandong Normal University（山东师范大学）；China Academy of Information and Communications Technology（中国信息通信研究院）；Central South University（中南大学）；Nanjing University of Science and Technology（南京理工大学）；Hebei University of Technology（河北工业大学）；Shandong University of Finance and Economics（山东财经大学）；National University of Defense Technology（国防科技大学）
University of Electronic Science and Technology of China（电子科技大学）；Cornell University（康奈尔大学）
The wavelet frame systems have been playing an active role in image restoration and many other image processing fields over the past decades, owing to the good capability of sparsely approximating piece-wise smooth functions such as images. In this paper, we propose a novel wavelet frame based sparse recovery model called \textit{Support Driven Sparse Regularization} (SDSR) for image deblurring, where the partial support information of frame coefficients is attained via a self-learning strategy and exploited via the proposed truncated $\ell_0$ regularization. Moreover, the state-of-the-art ima…
MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）；School of Computer Science and Technology, Shenzhen Campus of Harbin Institute of Technology（哈尔滨工业大学深圳校区计算机科学与技术学院）；School of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）；Hong Kong University of Science and Technology（香港科技大学）
Data61-Commonwealth Scientific Industrial Research Organization (CSIRO)（Data61-联邦科学与工业研究组织）；Australian National University（澳大利亚国立大学）
Deep convolutional neural networks perform better on images containing spatially invariant degradations, also known as synthetic degradations; however, their performance is limited on real-degraded photographs and requires multiple-stage network modeling. To advance the practicability of restoration algorithms, this paper proposes a novel single-stage blind real image restoration network (R$^2$Net) by employing a modular architecture. We use a residual on the residual structure to ease the flow of low-frequency information and apply feature attention to exploit the channel dependencies. Furthe…
As a widely studied task, video restoration aims to enhance the quality of the videos with multiple potential degradations, such as noises, blurs and compression artifacts. Among video restorations, compressed video quality enhancement and video super-resolution are two of the main tacks with significant values in practical scenarios. Recently, recurrent neural networks and transformers attract increasing research interests in this field, due to their impressive capability in sequence-to-sequence modeling. However, the training of these models is not only costly but also relatively hard to con…
Simon Fraser University（西蒙弗雷泽大学）；The University of Arizona（亚利桑那大学）
The theory of (tight) wavelet frames has been extensively studied in the past twenty years and they are currently widely used for image restoration and other image processing and analysis problems. The success of wavelet frame based models, including balanced approach and analysis based approach, is due to their capability of sparsely approximating piecewise smooth functions like images. Motivated by the balanced approach and analysis based approach, we shall propose a wavelet frame based $\ell_0$ minimization model, where the $\ell_0$ "norm" of the frame coefficients is penalized. We adapt th…
Deep LoRA-Unfolding Networks for Image Restoration
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（计算机科学与技术学院，哈尔滨工业大学（深圳））；Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（图像通信与网络工程院，上海交通大学）；School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院，南京大学）；School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen)（电子与信息工程学院，哈尔滨工业大学（深圳））
Deep-learning-based hyperspectral image (HSI) restoration methods have gained great popularity for their remarkable performance but often demand expensive network retraining whenever the specifics of task changes. In this paper, we propose to restore HSIs in a unified approach with an effective plug-and-play method, which can jointly retain the flexibility of optimization-based methods and utilize the powerful representation capability of deep neural networks. Specifically, we first develop a new deep HSI denoiser leveraging gated recurrent convolution units, short- and long-term skip connecti…
Shandong University of Traditional Chinese Medicine（山东中医药大学）；Nankai University（南开大学）；ShanghaiTech University（上海科技大学）；Taishan University（泰山学院）；United Imaging Co.（联影公司）
Real-world face super-resolution (SR) is a highly ill-posed image restoration task. The fully-cycled Cycle-GAN architecture is widely employed to achieve promising performance on face SR, but prone to produce artifacts upon challenging cases in real-world scenarios, since joint participation in the same degradation branch will impact final performance due to huge domain gap between real-world and synthetic LR ones obtained by generators. To better exploit the powerful generative capability of GAN for real-world face SR, in this paper, we establish two independent degradation branches in the fo…
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Hong Kong University of Science and Technology（香港科技大学）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Samsung Research China - Beijing (SRC-B)（三星中国研究院北京分部）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳校区）；SAMSUNG Electronics（三星电子）
Deep Variational Network Toward Blind Image Restoration
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算学系）；Macau Institute of Systems Engineering, Macau University of Science and Technology（澳门科技大学澳门系统工程研究所）
Beijing University of Technology（北京工业大学）；Beijing Key Laboratory of Computational Intelligence and Intelligent System（计算智能与智能系统北京市重点实验室）；The Hong Kong Polytechnic University（香港理工大学）
Dynamic scene deblurring is a challenging problem in computer vision. It is difficult to accurately estimate the spatially varying blur kernel by traditional methods. Data-driven-based methods usually employ kernel-free end-to-end mapping schemes, which are apt to overlook the kernel estimation. To address this issue, we propose a blur-attention module to dynamically capture the spatially varying features of non-uniform blurred images. The module consists of a DenseBlock unit and a spatial attention unit with multi-pooling feature fusion, which can effectively extract complex spatially varying…
