In this paper, we study design of deep neural networks for tasks of image restoration. We propose a novel style of residual connections dubbed "dual residual connection", which exploits the potential of paired operations, e.g., up- and down-sampling or convolution with large- and small-size kernels. We design a modular block implementing this connection style; it is equipped with two containers to which arbitrary paired operations are inserted. Adopting the "unraveled" view of the residual networks proposed by Veit et al., we point out that a stack of the proposed modular blocks allows the first operation in a block interact with the second operation in any subsequent blocks. Specifying the two operations in each of the stacked blocks, we build a complete network for each individual task of image restoration. We experimentally evaluate the proposed approach on five image restoration tasks using nine datasets. The results show that the proposed networks with properly chosen paired operations outperform previous methods on almost all of the tasks and datasets.
In many applications of deep learning, particularly those in image restoration, it is either very difficult, prohibitively expensive, or outright impossible to obtain paired training data precisely as in the real world. In such cases, one is forced to use synthesized paired data to train the deep convolutional neural network (DCNN). However, due to the unavoidable generalization error in statistical learning, the synthetically trained DCNN often performs poorly on real world data. To overcome this problem, we propose a new general training method that can compensate for, to a large extent, the…
Bio Imaging Signal Processing Lab Korea Ad. Inst. of Science & Technology (KAIST)（韩国科学技术院生物影像信号处理实验室）
The latest deep learning approaches perform better than the state-of-the-art signal processing approaches in various image restoration tasks. However, if an image contains many patterns and structures, the performance of these CNNs is still inferior. To address this issue, here we propose a novel feature space deep residual learning algorithm that outperforms the existing residual learning. The main idea is originated from the observation that the performance of a learning algorithm can be improved if the input and/or label manifolds can be made topologically simpler by an analytic mapping to …
CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
School of Geodesy and Geomatics, Wuhan University（武汉大学测绘学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditions
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；Rakuten Institute of Technology（乐天理工研究院）；Imperial College London（伦敦帝国学院）；KAIST（韩国科学技术院）；Tencent（腾讯）；Australian National University（澳大利亚国立大学）；Nanjing University（南京大学）
In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention b…
Residual Diffusion Bridge Model for Image Restoration
School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Zhongguancun Academy（中关村学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Robotics, Wuhan University（武汉大学机器人学院）
University of Electronic Science and Technology of China（电子科技大学）；Nokia Bell Labs（诺基亚贝尔实验室）；Nanyang Technological University（南洋理工大学）；University of Macau（澳门大学）；Nanjing Institute of Technology（南京工程学院）
In this paper, we propose a novel approach to the rank minimization problem, termed rank residual constraint (RRC) model. Different from existing low-rank based approaches, such as the well-known nuclear norm minimization (NNM) and the weighted nuclear norm minimization (WNNM), which estimate the underlying low-rank matrix directly from the corrupted observations, we progressively approximate the underlying low-rank matrix via minimizing the rank residual. Through integrating the image nonlocal self-similarity (NSS) prior with the proposed RRC model, we apply it to image restoration tasks, inc…
Northeastern University（东北大学）；University of Rochester（罗切斯特大学）；Rochester Institute of Technology（罗切斯特理工学院）；Huaqiao University（华侨大学）
Convolutional neural network has recently achieved great success for image restoration (IR) and also offered hierarchical features. However, most deep CNN based IR models do not make full use of the hierarchical features from the original low-quality images, thereby achieving relatively-low performance. In this paper, we propose a novel residual dense network (RDN) to address this problem in IR. We fully exploit the hierarchical features from all the convolutional layers. Specifically, we propose residual dense block (RDB) to extract abundant local features via densely connected convolutional …
SkillIR: Evolving Scene-Aware Skills for Agentic Image Restoration
Zhongnan University of Economics and Law（中南财经政法大学）；Wuhan University（武汉大学）；Dongguan University of Technology（东莞理工学院）；National Institute of Natural Hazards, Ministry of Emergency Management of China（中国应急管理部国家自然灾害防治研究院）
SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder
Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Hisilicon（海思半导体）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（雁栖湖北京数学科学与应用研究院）；State Key Laboratory of Membrane Biology, School of Life Sciences, Tsinghua University（清华大学生命科学学院膜生物学国家重点实验室）
Deep convolutional neural networks (DCNN) have been widely adopted for research on super resolution recently, however previous work focused mainly on stacking as many layers as possible in their model, in this paper, we present a new perspective regarding to image restoration problems that we can construct the neural network model reflecting the physical significance of the image restoration process, that is, embedding the a priori knowledge of image restoration directly into the structure of our neural network model, we employed a symmetric non-linear colorspace, the sigmoidal transfer, to re…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Interactive image restoration aims to generate restored images by adjusting a controlling coefficient which determines the restoration level. Previous works are restricted in modulating image with a single coefficient. However, real images always contain multiple types of degradation, which cannot be well determined by one coefficient. To make a step forward, this paper presents a new problem setup, called multi-dimension (MD) modulation, which aims at modulating output effects across multiple degradation types and levels. Compared with the previous single-dimension (SD) modulation, the MD is …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Bytedance Inc.（字节跳动公司）
While scale-invariant modeling has substantially boosted the performance of visual recognition tasks, it remains largely under-explored in deep networks based image restoration. Naively applying those scale-invariant techniques (e.g. multi-scale testing, random-scale data augmentation) to image restoration tasks usually leads to inferior performance. In this paper, we show that properly modeling scale-invariance into neural networks can bring significant benefits to image restoration performance. Inspired from spatial-wise convolution for shift-invariance, "scale-wise convolution" is proposed …
Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration
Southern University of Science and Technology（南方科技大学）；Shenzhen University（深圳大学）；Shenzhen University of Advanced Technology（深圳大学先进技术学院）；Xi’an Jiaotong University（西安交通大学）
Taiyuan University of Technology（太原理工大学）；Durham University（杜伦大学）
Image restoration methods are commonly used to improve the quality of astronomical images. In recent years, developments of deep neural networks and increments of the number of astronomical images have evoked a lot of data--driven image restoration methods. However, most of these methods belong to supervised learning algorithms, which require paired images either from real observations or simulated data as training set. For some applications, it is hard to get enough paired images from real observations and simulated images are quite different from real observed ones. In this paper, we propose…
Wavelet-Enhanced Desnowing: A Novel Single Image Restoration Approach for Traffic Surveillance under Adverse Weather Conditions
Automatic Control Research Institute, China Nuclear Power Engineering Co., Ltd.（中国核电工程有限公司自动控制研究院）；CNNC Engineering Research Center for Fuel Reprocessing（中核集团燃料后处理工程研究中心）；Chinese Aeronautical Establishment, Aviation Industry Corporation of China, Ltd（中国航空工业集团有限公司中国航空研究院）
M2IR: Proactive All-in-One Image Restoration via Mamba-style Modulation and Mixture-of-Experts
School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Computer Science and Technology, Anhui University of Technology（安徽工业大学计算机科学与技术学院）；Shenzhen Ubiquitous Data Enabling Key Laboratory, Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院深圳 ubiquitous 数据赋能重点实验室）；College of Artificial Intelligence, Taiyuan University of Technology（太原科技大学人工智能学院）
Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise
College of Computer Science, Nankai University（南开大学计算机学院）；School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）；Haihe Lab of ITAI（海河人工智能研究院）；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）
The principal rank-one (RO) components of an image represent the self-similarity of the image, which is an important property for image restoration. However, the RO components of a corrupted image could be decimated by the procedure of image denoising. We suggest that the RO property should be utilized and the decimation should be avoided in image restoration. To achieve this, we propose a new framework comprised of two modules, i.e., the RO decomposition and RO reconstruction. The RO decomposition is developed to decompose a corrupted image into the RO components and residual. This is achieve…
Digital scans of analogue photographic film typically contain artefacts such as dust and scratches. Automated removal of these is an important part of preservation and dissemination of photographs of historical and cultural importance. While state-of-the-art deep learning models have shown impressive results in general image inpainting and denoising, film artefact removal is an understudied problem. It has particularly challenging requirements, due to the complex nature of analogue damage, the high resolution of film scans, and potential ambiguities in the restoration. There are no publicly av…
Benchmarking Endoscopic Surgical Image Restoration and Beyond
The Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；Southern Medical University（南方医科大学）；Xidian University（西安电子科技大学）；Wuhan University（武汉大学）
Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Hisilicon（海思）；Institute for Interdisciplinary Information Science, Tsinghua University（清华大学交叉信息研究院）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（北京雁栖湖应用数学研究院）
Collecting paired training data is difficult in practice, but the unpaired samples broadly exist. Current approaches aim at generating synthesized training data from unpaired samples by exploring the relationship between the corrupted and clean data. This work proposes LUD-VAE, a deep generative method to learn the joint probability density function from data sampled from marginal distributions. Our approach is based on a carefully designed probabilistic graphical model in which the clean and corrupted data domains are conditionally independent. Using variational inference, we maximize the evi…
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Tencent（腾讯）；The Chinese University of Hong Kong at Shenzhen（香港中文大学（深圳））
Restoring the clean background from the superimposed images containing a noisy layer is the common crux of a classical category of tasks on image restoration such as image reflection removal, image deraining and image dehazing. These tasks are typically formulated and tackled individually due to the diverse and complicated appearance patterns of noise layers within the image. In this work we present the Deep-Masking Generative Network (DMGN), which is a unified framework for background restoration from the superimposed images and is able to cope with different types of noise. Our proposed DMGN…
Beijing Institute of Technology（北京理工大学）；National Institute of Informatics（国立信息学研究所）
A series of methods have been proposed to reconstruct an image from compressively sensed random measurement, but most of them have high time complexity and are inappropriate for patch-based compressed sensing capture, because of their serious blocky artifacts in the restoration results. In this paper, we present a non-iterative image reconstruction method from patch-based compressively sensed random measurement. Our method features two cascaded networks based on residual convolution neural network to learn the end-to-end full image restoration, which is capable of reconstructing image patches …
SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration
School of Big Data and Software Engineering, Chongqing University（大数据与软件工程学院，重庆大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Microelectronics and Communication Engineering, Chongqing University（微电子与通信工程学院，重庆大学）；School of Electrical and Information Engineering, Tianjin University（电气与信息工程学院，天津大学）
Unsupervised Variational Translator for Bridging Image Restoration and High-Level Vision Tasks
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）
Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from low-quality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper, we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer. SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image recons…
Data61-Commonwealth Scientific Industrial Research Organization (CSIRO)（Data61-联邦科学与工业研究组织）；Australian National University（澳大利亚国立大学）
Deep convolutional neural networks perform better on images containing spatially invariant degradations, also known as synthetic degradations; however, their performance is limited on real-degraded photographs and requires multiple-stage network modeling. To advance the practicability of restoration algorithms, this paper proposes a novel single-stage blind real image restoration network (R$^2$Net) by employing a modular architecture. We use a residual on the residual structure to ease the flow of low-frequency information and apply feature attention to exploit the channel dependencies. Furthe…
Underwater Image Restoration Through a Prior Guided Hybrid Sense Approach and Extensive Benchmark Analysis
University of Macau（澳门大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；School of Big Data, Baoshan University（保山学院大数据学院）；School of Computer Science and Engineering, Huizhou University（惠州学院计算机科学与工程学院）
提出 Color Balance Prior 引导的多尺度混合感知框架 GuidedHybSensUIR，并构建综合水下复原基准，实验显示其总体优于37种现有方法。
Image demosaicing is problem of interpolating full- resolution color images from raw sensor (color filter array) data. During last decade, deep neural networks have been widely used in image restoration, and in particular, in demosaicing, attaining significant performance improvement. In recent years, vision transformers have been designed and successfully used in various computer vision applications. One of the recent methods of image restoration based on a Swin Transformer (ST), SwinIR, demonstrates state-of-the-art performance with a smaller number of parameters than neural network-based me…
Tsinghua University（清华大学）；ETH Zürich（苏黎世联邦理工学院）；ByteDance Inc（字节跳动公司）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Würzburg（维尔茨堡大学）
Lighter and faster image restoration (IR) models are crucial for the deployment on resource-limited devices. Binary neural network (BNN), one of the most promising model compression methods, can dramatically reduce the computations and parameters of full-precision convolutional neural networks (CNN). However, there are different properties between BNN and full-precision CNN, and we can hardly use the experience of designing CNN to develop BNN. In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks.…
TIDE: Two-Stage Inverse Degradation Estimation with Guided Prior Disentanglement for Underwater Image Restoration
Mohamed bin Zayed University of AI（莫扎德·本·泽亚德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Nagasaki University（长崎大学）
