Image enhancement from degradation of rainy artifacts plays a critical role in outdoor visual computing systems. In this paper, we tackle the notion of scale that deals with visual changes in appearance of rain steaks with respect to the camera. Specifically, we revisit multi-scale representation by scale-space theory, and propose to represent the multi-scale correlation in convolutional feature domain, which is more compact and robust than that in pixel domain. Moreover, to improve the modeling ability of the network, we do not treat the extracted multi-scale features equally, but design a novel scale-space invariant attention mechanism to help the network focus on parts of the features. In this way, we summarize the most activated presence of feature maps as the salient features. Extensive experiments results on synthetic and real rainy scenes demonstrate the superior performance of our scheme over the state-of-the-arts.
Nanjing University of Science and Technology（南京理工大学）；China Electronics Technology Group Corporation（中国电子科技集团有限公司）
Transformers-based methods have achieved significant performance in image deraining as they can model the non-local information which is vital for high-quality image reconstruction. In this paper, we find that most existing Transformers usually use all similarities of the tokens from the query-key pairs for the feature aggregation. However, if the tokens from the query are different from those of the key, the self-attention values estimated from these tokens also involve in feature aggregation, which accordingly interferes with the clear image restoration. To overcome this problem, we propose …
Unpaired Image Deraining Using Reward-Guided Self-Reinforcement Strategy
National University of Defense Technology（国防科技大学）；National University of Singapore（新加坡国立大学）；Nanjing University of Science and Technology（南京理工大学）；Hefei University of Technology（合肥工业大学）；University of Trento（特伦斯大学）
Australian National University（澳大利亚国立大学）；Data61-CSIRO（澳大利亚联邦科学与工业研究组织数据61研究所）
Over parameterization is a common technique in deep learning to help models learn and generalize sufficiently to the given task; nonetheless, this often leads to enormous network structures and consumes considerable computing resources during training. Recent powerful transformer-based deep learning models on vision tasks usually have heavy parameters and bear training difficulty. However, many dense-prediction low-level computer vision tasks, such as rain streak removing, often need to be executed on devices with limited computing power and memory in practice. Hence, we introduce a recursive …
Cross Paradigm Representation and Alignment Transformer for Image Deraining
Nanjing Agricultural University（南京农业大学）；Soochow University（苏州大学）；Xiangtan University（湘潭大学）；Shanghai University（上海大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Westlake University（西湖大学）
Dalian University of Technology（大连理工大学）；Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province（辽宁省计算数学与数据智能重点实验室）；University of Macau（澳门大学）
In the field of multimedia, single image deraining is a basic pre-processing work, which can greatly improve the visual effect of subsequent high-level tasks in rainy conditions. In this paper, we propose an effective algorithm, called JDNet, to solve the single image deraining problem and conduct the segmentation and detection task for applications. Specifically, considering the important information on multi-scale features, we propose a Scale-Aggregation module to learn the features with different scales. Simultaneously, Self-Attention module is introduced to match or outperform their convol…
GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditions
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；Rakuten Institute of Technology（乐天理工研究院）；Imperial College London（伦敦帝国学院）；KAIST（韩国科学技术院）；Tencent（腾讯）；Australian National University（澳大利亚国立大学）；Nanjing University（南京大学）
Zhengzhou University（郑州大学）；University of Technology Sydney（悉尼科技大学）
Single image deraining (SID) is an important and challenging topic in emerging vision applications, and most of emerged deraining methods are supervised relying on the ground truth (i.e., paired images) in recent years. However, in practice it is rather common to have no un-paired images in real deraining task, in such cases how to remove the rain streaks in an unsupervised way will be a very challenging task due to lack of constraints between images and hence suffering from low-quality recovery results. In this paper, we explore the unsupervised SID task using unpaired data and propose a nove…
Wuhan University（武汉大学）；University of Central Florida（中佛罗里达大学）；Shenzhen University（深圳大学）；National Tsing Hua University（台湾清华大学）
Convolutional neural network (CNN) and Transformer have achieved great success in multimedia applications. However, little effort has been made to effectively and efficiently harmonize these two architectures to satisfy image deraining. This paper aims to unify these two architectures to take advantage of their learning merits for image deraining. In particular, the local connectivity and translation equivariance of CNN and the global aggregation ability of self-attention (SA) in Transformer are fully exploited for specific local context and global structure representations. Based on the obser…
Recently, image restoration transformers have achieved comparable performance with previous state-of-the-art CNNs. However, how to efficiently leverage such architectures remains an open problem. In this work, we present Dual-former whose critical insight is to combine the powerful global modeling ability of self-attention modules and the local modeling ability of convolutions in an overall architecture. With convolution-based Local Feature Extraction modules equipped in the encoder and the decoder, we only adopt a novel Hybrid Transformer Block in the latent layer to model the long-distance d…
University of Science and Technology of China（中国科学技术大学）
We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, zero-shot restoration, and hard patch refinement. Using the pre-trained transformers, our model is able to tell the motion difference between the true image information and the obstructing elements. For zero-shot image restoration, we design a novel model, termed SiamTrans, which is constructed by Siamese transformers, encoders, and decoders. Each transformer has a te…
Australian National University（澳大利亚国立大学）；Tencent AI Laboratory（腾讯人工智能实验室）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Rain streaks and rain drops are two natural phenomena, which degrade image capture in different ways. Currently, most existing deep deraining networks take them as two distinct problems and individually address one, and thus cannot deal adequately with both simultaneously. To address this, we propose a Dual Attention-in-Attention Model (DAiAM) which includes two DAMs for removing both rain streaks and raindrops. Inside the DAM, there are two attentive maps - each of which attends to the heavy and light rainy regions, respectively, to guide the deraining process differently for applicable regio…
Exploiting Regional Information Transformer for Single Image Deraining
The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Joint multi-dimensional dynamic attention and transformer for general image restoration
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；Key Laboratory of Photonic Technology for Integrated Sensing and Communication, Ministry of Education（教育部光子技术集成感知与通信重点实验室）；Guangdong Provincial Key Laboratory of Information Photonics Technology（广东省信息光子技术重点实验室）；Institute of Advanced Photonics Technology, School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院先进光子技术研究所）；School of Electronics and Communication Engineering, Sun Yat-Sen University（中山大学电子与通信工程学院）
Images captured under complicated rain conditions often suffer from noticeable degradation of visibility. The rain models generally introduce diversity visibility degradation, which includes rain streak, rain drop as well as rain mist. Numerous existing single image deraining methods focus on the only one type rain model, which does not have strong generalization ability. In this paper, we propose a novel end-to-end Neuron Attention Stage-by-Stage Net (NASNet), which can solve all types of rain model tasks efficiently. For one thing, we pay more attention on the Neuron relationship and propose…
FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining
Shandong University（山东大学）；Central South University（中南大学）；Nanjing University of Science and Technology（南京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；National University of Defense Technology（国防科技大学）
University of Electronic Science and Technology of China（电子科技大学）；The Shanghai Jiaotong University（上海交通大学）
Different rain models and novel network structures have been proposed to remove rain streaks from single rainy images. In this work, we bring attention to the intrinsic priors and multi-scale features of the rainy images, and develop several intrinsic loss functions to train a CNN deraining network. We first study the sparse priors of rainy images, which have been verified to preserve unbroken edges in image decomposition. However, its mathematical formulation usually leads to an intractable solution, we propose quasi-sparsity priors to decrease complexity, so that our network can be trained u…
ASUS Intelligent Cloud Services（华硕智能云服务）；Georgia Institute of Technology（佐治亚理工学院）；National Taiwan University（台湾大学）
Single image deraining is a crucial problem because rain severely degenerates the visibility of images and affects the performance of computer vision tasks like outdoor surveillance systems and intelligent vehicles. In this paper, we propose the new convolutional neural network (CNN) called the wavelet channel attention module with a fusion network. Wavelet transform and the inverse wavelet transform are substituted for down-sampling and up-sampling so feature maps from the wavelet transform and convolutions contain different frequencies and scales. Furthermore, feature maps are integrated by …
Wuhan University of Science and Technology（武汉科技大学）
It is challenging to remove rain-steaks from a single rainy image because the rain steaks are spatially varying in the rainy image. This problem is studied in this paper by combining conventional image processing techniques and deep learning based techniques. An improved weighted guided image filter (iWGIF) is proposed to extract high frequency information from a rainy image. The high frequency information mainly includes rain steaks and noise, and it can guide the rain steaks aware deep convolutional neural network (RSADCNN) to pay more attention to rain steaks. The efficiency and explain-abi…
University of Science and Technology of China（中国科学技术大学）；University of Macau（澳门大学）；University of Chinese Academy of Sciences（中国科学院大学）
In this paper, we present Uformer, an effective and efficient Transformer-based architecture for image restoration, in which we build a hierarchical encoder-decoder network using the Transformer block. In Uformer, there are two core designs. First, we introduce a novel locally-enhanced window (LeWin) Transformer block, which performs nonoverlapping window-based self-attention instead of global self-attention. It significantly reduces the computational complexity on high resolution feature map while capturing local context. Second, we propose a learnable multi-scale restoration modulator in the…
University of Science and Technology of China（中国科学技术大学）；Huawei Technologies（华为技术有限公司）；Huawei Cloud BU（华为云业务单元）
When smartphone cameras are used to take photos of digital screens, usually moire patterns result, severely degrading photo quality. In this paper, we design a wavelet-based dual-branch network (WDNet) with a spatial attention mechanism for image demoireing. Existing image restoration methods working in the RGB domain have difficulty in distinguishing moire patterns from true scene texture. Unlike these methods, our network removes moire patterns in the wavelet domain to separate the frequencies of moire patterns from the image content. The network combines dense convolution modules and dilate…
A recent line of convolutional neural network-based works has succeeded in capturing rain streaks. However, difficulties in detailed recovery still remain. In this paper, we present a multi-level connection and wide regional non-local block network (MCW-Net) to properly restore the original background textures in rainy images. Unlike existing encoder-decoder-based image deraining models that improve performance with additional branches, MCW-Net improves performance by maximizing information utilization without additional branches through the following two proposed methods. The first method is …
Dalian University of Technology（大连理工大学）；City University of Hong Kong（香港城市大学）
Removing rain streaks from a single image has been drawing considerable attention as rain streaks can severely degrade the image quality and affect the performance of existing outdoor vision tasks. While recent CNN-based derainers have reported promising performances, deraining remains an open problem for two reasons. First, existing synthesized rain datasets have only limited realism, in terms of modeling real rain characteristics such as rain shape, direction and intensity. Second, there are no public benchmarks for quantitative comparisons on real rain images, which makes the current evalua…
A Hybrid Transformer-Mamba Network for Single Image Deraining
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间科学与技术学院）；Yunnan Normal University（云南师范大学）
Intra and Inter Parser-Prompted Transformers for Effective Image Restoration
Sun Yat-sen University（中山大学）；Centre for Advances in Reliability and Safety（可靠性与安全先进中心）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Science and Technology（南京理工大学）；Dalian University of Technology（大连理工大学）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
How to aggregate spatial information plays an essential role in learning-based image restoration. Most existing CNN-based networks adopt static convolutional kernels to encode spatial information, which cannot aggregate spatial information adaptively. Recent transformer-based architectures achieve adaptive spatial aggregation. But they lack desirable inductive biases of convolutions and require heavy computational costs. In this paper, we propose a kernel basis attention (KBA) module, which introduces learnable kernel bases to model representative image patterns for spatial information aggrega…
Single image deraining is typically addressed as residual learning to predict the rain layer from an input rainy image. For this purpose, an encoder-decoder network draws wide attention, where the encoder is required to encode a high-quality rain embedding which determines the performance of the subsequent decoding stage to reconstruct the rain layer. However, most of existing studies ignore the significance of rain embedding quality, thus leading to limited performance with over/under-deraining. In this paper, with our observation of the high rain layer reconstruction performance by an rain-t…
Boosting Image Restoration via Priors from Pre-trained Models
Zhejiang Lab（之江实验室）；CUHK（香港中文大学）；RealityEdge（睿迪智芯）；University of Macau（澳门大学）；Institute of Collaborative Innovation（协同创新研究院）；Texas A&M University（德克萨斯农工大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）
LoopMoEVR: Loop-Based Degradation-Aware Mixture-of-Experts for Unified UHD Video Restoration
Shandong Normal University（山东师范大学）；China Academy of Information and Communications Technology（中国信息通信研究院）；Central South University（中南大学）；Nanjing University of Science and Technology（南京理工大学）；Hebei University of Technology（河北工业大学）；Shandong University of Finance and Economics（山东财经大学）；National University of Defense Technology（国防科技大学）
Single image deraining (SID) in real scenarios attracts increasing attention in recent years. Due to the difficulty in obtaining real-world rainy/clean image pairs, previous real datasets suffer from low-resolution images, homogeneous rain streaks, limited background variation, and even misalignment of image pairs, resulting in incomprehensive evaluation of SID methods. To address these issues, we establish a new high-quality dataset named RealRain-1k, consisting of $1,120$ high-resolution paired clean and rainy images with low- and high-density rain streaks, respectively. Images in RealRain-1…
Whiting School of Engineering, Johns Hopkins University（约翰斯·霍普金斯大学怀廷工程学院）
Removal of rain streaks from a single image is an extremely challenging problem since the rainy images often contain rain streaks of different size, shape, direction and density. Most recent methods for deraining use a deep network following a generic "encoder-decoder" architecture which captures low-level features across the initial layers and high-level features in the deeper layers. For the task of deraining, the rain streaks which are to be removed are relatively small and focusing much on global features is not an efficient way to solve the problem. To this end, we propose using an overco…
Inception Institute of AI（人工智能起源研究所）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to ap…
Zhejiang University of Technology（浙江工业大学）；Dalian University of Technology（大连理工大学）；Pazhou Lab（琶洲实验室）
Video deraining is an important issue for outdoor vision systems and has been investigated extensively. However, designing optimal architectures by the aggregating model formation and data distribution is a challenging task for video deraining. In this paper, we develop a model-guided triple-level optimization framework to deduce network architecture with cooperating optimization and auto-searching mechanism, named Triple-level Model Inferred Cooperating Searching (TMICS), for dealing with various video rain circumstances. In particular, to mitigate the problem that existing methods cannot cov…
Inception Institute of AI（Inception人工智能研究所）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch t…
Vietnam National Space Center（越南国家航天中心）；Vietnam Academy of Science and Technology（越南科学技术研究院）；Gachon University（嘉泉大学）
Image deraining is a new challenging problem in applications of autonomous vehicles. In a bad weather condition of heavy rainfall, raindrops, mainly hitting the vehicle's windshield, can significantly reduce observation ability even though the windshield wipers might be able to remove part of it. Moreover, rain flows spreading over the windshield can yield the physical effect of refraction, which seriously impede the sightline or undermine the machine learning system equipped in the vehicle. In this paper, we propose a new multi-stage multi-task recurrent generative adversarial network (M2GAN)…
