Restormer: Efficient Transformer for High-Resolution Image Restoration
Inception Institute of AI（人工智能起源研究所）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang
Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.
A Comparative Study of Image Restoration Networks for General Backbone Network Design
University of Macau（澳门大学）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）
Beyond Model Design: Data-Centric Training and Self-Ensemble for Gaussian Color Image Denoising
Hefei University of Technology（合肥工业大学）；Hangzhou Dianzi University（杭州电子科技大学）；Jinan University（暨南大学）；South China Agricultural University（华南农业大学）；The University of Tokyo（东京大学）
An Adaptive Edge-Guided Dual-Network Framework for Fast QR Code Motion Deblurring
Guangdong Provincial Key Lab of Robotics and Intelligent Systems（广东省机器人与智能系统重点实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；School of Nano Science and Technology, University of Science and Technology of China（中国科学技术大学纳米科学与技术学院）；Shenzhen University of Advanced Technology（深圳大学）
Xinjiang University（新疆大学）；Shandong University of Technology（山东理工大学）
Although Convolutional Neural Networks (CNN) have made good progress in image restoration, the intrinsic equivalence and locality of convolutions still constrain further improvements in image quality. Recent vision transformer and self-attention have achieved promising results on various computer vision tasks. However, directly utilizing Transformer for image restoration is a challenging task. In this paper, we introduce an effective hybrid architecture for sand image restoration tasks, which leverages local features from CNN and long-range dependencies captured by transformer to improve the r…
While Transformer has achieved remarkable performance in various high-level vision tasks, it is still challenging to exploit the full potential of Transformer in image restoration. The crux lies in the limited depth of applying Transformer in the typical encoder-decoder framework for image restoration, resulting from heavy self-attention computation load and inefficient communications across different depth (scales) of layers. In this paper, we present a deep and effective Transformer-based network for image restoration, termed as U2-Former, which is able to employ Transformer as the core oper…
University of Science and Technology of China（中国科学技术大学）
We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a te…
University of Science and Technology of China（中国科学技术大学）；University of Macau（澳门大学）；University of Chinese Academy of Sciences（中国科学院大学）
In this paper, we present Uformer, an effective and efficient Transformer-based architecture for image restoration, in which we build a hierarchical encoder-decoder network using the Transformer block. In Uformer, there are two core designs. First, we introduce a novel locally-enhanced window (LeWin) Transformer block, which performs nonoverlapping window-based self-attention instead of global self-attention. It significantly reduces the computational complexity on high resolution feature map while capturing local context. Second, we propose a learnable multi-scale restoration modulator in the…
Beware of Aliases -- Signal Preservation is Crucial for Robust Image Restoration
University of Mannheim（曼海姆大学）；Institute for Machine Learning and Analytics（机器学习与分析研究所）；Offenburg University（奥芬堡大学）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；Saarland Informatics Campus（萨尔州信息校区）
Recently, image restoration transformers have achieved comparable performance with previous state-of-the-art CNNs. However, how to efficiently leverage such architectures remains an open problem. In this work, we present Dual-former whose critical insight is to combine the powerful global modeling ability of self-attention modules and the local modeling ability of convolutions in an overall architecture. With convolution-based Local Feature Extraction modules equipped in the encoder and the decoder, we only adopt a novel Hybrid Transformer Block in the latent layer to model the long-distance d…
Shanghai Jiao Tong University（上海交通大学）；ETH Zürich（苏黎世联邦理工学院）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Westlake University（西湖大学）
Recently, Transformer architecture has been introduced into image restoration to replace convolution neural network (CNN) with surprising results. Considering the high computational complexity of Transformer with global attention, some methods use the local square window to limit the scope of self-attention. However, these methods lack direct interaction among different windows, which limits the establishment of long-range dependencies. To address the above issue, we propose a new image restoration model, Cross Aggregation Transformer (CAT). The core of our CAT is the Rectangle-Window Self-Att…
GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditions
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；Rakuten Institute of Technology（乐天理工研究院）；Imperial College London（伦敦帝国学院）；KAIST（韩国科学技术院）；Tencent（腾讯）；Australian National University（澳大利亚国立大学）；Nanjing University（南京大学）
Due to various and complicated snow degradations, single image desnowing is a challenging image restoration task. As prior arts can not handle it ideally, we propose a novel transformer, SnowFormer, which explores efficient cross-attentions to build local-global context interaction across patches and surpasses existing works that employ local operators or vanilla transformers. Compared to prior desnowing methods and universal image restoration methods, SnowFormer has several benefits. Firstly, unlike the multi-head self-attention in recent image restoration Vision Transformers, SnowFormer inco…
Global operations, such as global average pooling, are widely used in top-performance image restorers. They aggregate global information from input features along entire spatial dimensions but behave differently during training and inference in image restoration tasks: they are based on different regions, namely the cropped patches (from images) and the full-resolution images. This paper revisits global information aggregation and finds that the image-based features during inference have a different distribution than the patch-based features during training. This train-test inconsistency negat…
TDiR: Transformer based Diffusion for Image Restoration Tasks
Department of Computer Vision（计算机视觉系）；FutureDataMinds；Electrical Engineering Dept（电气工程系）；King Fahd University of Petroleum and Minerals（国王法赫德石油和矿产大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Western Australia（西澳大学）；School of Computing（计算学院）
National University of Defense Technology（国防科技大学）；National Innovation Institute of Defense Technology（国防科技创新研究院）
The great success of Deep Neural Networks (DNNs) has inspired the algorithmic development of DNN-based Fixed-Point (DNN-FP) for computer vision tasks. DNN-FP methods, trained by Back-Propagation Through Time or computing the inaccurate inversion of the Jacobian, suffer from inferior representation ability. Motivated by the representation power of the Transformer, we propose a framework to unroll the FP and approximate each unrolled process via Transformer blocks, called FPformer. To reduce the high consumption of memory and computation, we come up with FPRformer by sharing parameters between t…
Shanghai Jiao Tong University（上海交通大学）；ETH Zürich（苏黎世联邦理工学院）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Westlake University（西湖大学）
Recently, Transformer-based image restoration networks have achieved promising improvements over convolutional neural networks due to parameter-independent global interactions. To lower computational cost, existing works generally limit self-attention computation within non-overlapping windows. However, each group of tokens are always from a dense area of the image. This is considered as a dense attention strategy since the interactions of tokens are restrained in dense regions. Obviously, this strategy could result in restricted receptive fields. To address this issue, we propose Attention Re…
Enhancing Image Restoration Transformer via Adaptive Translation Equivariance
Institute of Medical Technology, Peking University Health Science Center, Peking University（北京大学医学部医学技术研究所）；Biomedical Engineering Department, College of Future Technology, Peking University（未来技术学院生物医学工程系）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）；China Mobile Research Institute（中国移动研究院）
Joint multi-dimensional dynamic attention and transformer for general image restoration
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education（教育部光子技术集成感知与通信重点实验室）；Guangdong Provincial Key Laboratory of Information Photonics Technology（广东省信息光子技术重点实验室）；Institute of Advanced Photonics Technology, School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院先进光子技术研究所）；School of Electronics and Communication Engineering, Sun Yat-Sen University（中山大学电子与通信工程学院）
Intra and Inter Parser-Prompted Transformers for Effective Image Restoration
Sun Yat-sen University（中山大学）；Centre for Advances in Reliability and Safety（可靠性与安全先进中心）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）
In order to get raw images of high quality for downstream Image Signal Process (ISP), in this paper we present an Efficient Locally Multiplicative Transformer called ELMformer for raw image restoration. ELMformer contains two core designs especially for raw images whose primitive attribute is single-channel. The first design is a Bi-directional Fusion Projection (BFP) module, where we consider both the color characteristics of raw images and spatial structure of single-channel. The second one is that we propose a Locally Multiplicative Self-Attention (L-MSA) scheme to effectively deliver infor…
Multi-dimensional Visual Prompt Enhanced Image Restoration via Mamba-Transformer Aggregation
School of Digital Industry, Jiangxi Normal University（江西师范大学数字产业学院）；School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）
MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）；School of Computer Science and Technology, Shenzhen Campus of Harbin Institute of Technology（哈尔滨工业大学深圳校区计算机科学与技术学院）；School of Engineering and Computer Science, Australian National University（澳大利亚国立大学工程与计算机科学学院）；Hong Kong University of Science and Technology（香港科技大学）
HAT: Hybrid Attention Transformer for Image Restoration
State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学物联网智能城市国家重点实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）
How Powerful Potential of Attention on Image Restoration?
Sun Yat-sen University（中山大学）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Dalian University of Technology（大连理工大学）
ATD: Improved Transformer with Adaptive Token Dictionary for Image Restoration
School of Computer Science and Engineering (School of Cyber Security), University of Electronic Science and Technology of China (UESTC)（计算机科学与工程学院（网络安全学院），电子科学与技术大学（UESTC））
Look-Around Before You Leap: High-Frequency Injected Transformer for Image Restoration
College of Computer Science, Nankai University（南开大学计算机学院）；Nankai International Advanced Research Institute（南开大学国际高等研究院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Neural Discrimination-Prompted Transformers for Efficient UHD Image Restoration and Enhancement
Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）；University of California, San Francisco（加州大学旧金山分校）
Ultra-High-Definition Restoration Transformers with Correlation Matching Transformation
University of California, San Francisco（加利福尼亚大学旧金山分校）；School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computing and Communications, Lancaster University（兰卡斯特大学计算与通信学院）
Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration
College of Computer Science, Nankai University（南开大学计算机学院）；Nankai International Advanced Research Institute（南开大学国际高等研究院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Sharing Key Semantics in Transformer Makes Efficient Image Restoration
ETH Zürich（苏黎世联邦理工学院）；INSAIT, Sofia University（保加利亚索菲亚大学INSAIT研究院）；University of Pisa（比萨大学）；University of Trento（特伦托大学）；Meta Reality Labs（元宇宙现实实验室）；Peking University（北京大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of California, Merced（加州大学默塞德分校）
Correlation Matching Transformation Transformers for UHD Image Restoration
The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）；National University of Singapore（新加坡国立大学）；Hebei University of Technology（河北工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Training-Free Large Model Priors for Multiple-in-One Image Restoration
Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；Xiamen University（厦门大学）
