Rainy weather will have a significant impact on the regular operation of the imaging system. Based on this premise, image rain removal has always been a popular branch of low-level visual tasks, especially methods using deep neural networks. However, most neural networks are but-branched, such as only using convolutional neural networks or Transformers, which is unfavourable for the multidimensional fusion of image features. In order to solve this problem, this paper proposes a dual-branch attention fusion network. Firstly, a two-branch network structure is proposed. Secondly, an attention fusion module is proposed to selectively fuse the features extracted by the two branches rather than simply adding them. Finally, complete ablation experiments and sufficient comparison experiments prove the rationality and effectiveness of the proposed method.
Xiamen University（厦门大学）；University of Science and Technology of China（中国科学技术大学）；Columbia University（哥伦比亚大学）
We propose a simple yet effective deep tree-structured fusion model based on feature aggregation for the deraining problem. We argue that by effectively aggregating features, a relatively simple network can still handle tough image deraining problems well. First, to capture the spatial structure of rain we use dilated convolutions as our basic network block. We then design a tree-structured fusion architecture which is deployed within each block (spatial information) and across all blocks (content information). Our method is based on the assumption that adjacent features contain redundant info…
Shenyang Aerospace University（沈阳航空航天大学）；Shenyang Fire Science and Technology Research Institute of MEM（沈阳消防科学研究所）
Rain streaks bring serious blurring and visual quality degradation, which often vary in size, direction and density. Current CNN-based methods achieve encouraging performance, while are limited to depict rain characteristics and recover image details in the poor visibility environment. To address these issues, we present a Multi-scale Hourglass Hierarchical Fusion Network (MH2F-Net) in end-to-end manner, to exactly captures rain streak features with multi-scale extraction, hierarchical distillation and information aggregation. For better extracting the features, a novel Multi-scale Hourglass E…
Wuhan University（武汉大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；King’s College London（伦敦国王学院）；Harbin Institute of Technology（哈尔滨工业大学）
Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image…
ASUS Intelligent Cloud Services（华硕智能云服务）；Georgia Institute of Technology（佐治亚理工学院）；National Taiwan University（台湾大学）
Single image deraining is a crucial problem because rain severely degenerates the visibility of images and affects the performance of computer vision tasks like outdoor surveillance systems and intelligent vehicles. In this paper, we propose the new convolutional neural network (CNN) called the wavelet channel attention module with a fusion network. Wavelet transform and the inverse wavelet transform are substituted for down-sampling and up-sampling so feature maps from the wavelet transform and convolutions contain different frequencies and scales. Furthermore, feature maps are integrated by …
Single image rain streaks removal is extremely important since rainy images adversely affect many computer vision systems. Deep learning based methods have found great success in image deraining tasks. In this paper, we propose a novel residual-guide feature fusion network, called ResGuideNet, for single image deraining that progressively predicts highquality reconstruction. Specifically, we propose a cascaded network and adopt residuals generated from shallower blocks to guide deeper blocks. By using this strategy, we can obtain a coarse to fine estimation of negative residual as the blocks g…
Hefei University of Technology（合肥工业大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Single image deraining task is still a very challenging task due to its ill-posed nature in reality. Recently, researchers have tried to fix this issue by training the CNN-based end-to-end models, but they still cannot extract the negative rain streaks from rainy images precisely, which usually leads to an over de-rained or under de-rained result. To handle this issue, this paper proposes a new coarse-to-fine single image deraining framework termed Multi-stream Hybrid Deraining Network (shortly, MH-DerainNet). To obtain the negative rain streaks during training process more accurately, we pres…
We present the Cross-stitched Multi-task Unified Dual Recursive Network (CMUDRN) model targeting the task of unified deraining and desnowing in a multi-task learning setting. This unified model borrows from the basic Dual Recursive Network (DRN) architecture developed by Cai et al. The proposed model makes use of cross-stitch units that enable multi-task learning across two separate DRN models, each tasked for single image deraining and desnowing, respectively. By fixing cross-stitch units at several layers of basic task-specific DRN networks, we perform multi-task learning over the two separa…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Inception Institute of Artificial Intelligence（启元人工智能研究院）；Meituan Group（美团集团）；Australian National University（澳大利亚国立大学）；Tencent（腾讯）；Beijing Institute of Technology（北京理工大学）
Rain is a common natural phenomenon. Taking images in the rain however often results in degraded quality of images, thus compromises the performance of many computer vision systems. Most existing de-rain algorithms use only one single input image and aim to recover a clean image. Few work has exploited stereo images. Moreover, even for single image based monocular deraining, many current methods fail to complete the task satisfactorily because they mostly rely on per pixel loss functions and ignore semantic information. In this paper, we present a Paired Rain Removal Network (PRRNet), which ex…
University of Electronic Science and Technology of China（电子科技大学）；Tencent AI Lab（腾讯人工智能实验室）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）
Single image deraining regards an input image as a fusion of a background image, a transmission map, rain streaks, and atmosphere light. While advanced models are proposed for image restoration (i.e., background image generation), they regard rain streaks with the same properties as background rather than transmission medium. As vapors (i.e., rain streaks accumulation or fog-like rain) are conveyed in the transmission map to model the veiling effect, the fusion of rain streaks and vapors do not naturally reflect the rain image formation. In this work, we reformulate rain streaks as transmissio…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Ryerson University（瑞尔森大学）；Hefei University of Technology（合肥工业大学）
The intricacy of rainy image contents often leads cutting-edge deraining models to image degradation including remnant rain, wrongly-removed details, and distorted appearance. Such degradation is further exacerbated when applying the models trained on synthetic data to real-world rainy images. We observe two types of domain gaps between synthetic and real-world rainy images: one exists in rain streak patterns; the other is the pixel-level appearance of rain-free images. To bridge the two domain gaps, we propose a semi-supervised detail-recovery image deraining network (Semi-DRDNet) with dual s…
Shenyang Aerospace University（沈阳航空航天大学）；Nanjing University of Science and Technology（南京理工大学）；Wuhan University（武汉大学）
Learning single image deraining (SID) networks from an unpaired set of clean and rainy images is practical and valuable as acquiring paired real-world data is almost infeasible. However, without the paired data as the supervision, learning a SID network is challenging. Moreover, simply using existing unpaired learning methods (e.g., unpaired adversarial learning and cycle-consistency constraints) in the SID task is insufficient to learn the underlying relationship from rainy inputs to clean outputs as there exists significant domain gap between the rainy and clean images. In this paper, we dev…
Learning Dual-Domain Multi-Scale Representations for Single Image Deraining
Nanjing Agricultural University（南京农业大学）；Xiangtan University（湘潭大学）；Nanjing University（南京大学）；Northeast Forestry University（东北林业大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Westlake University（西湖大学）
We develop a new physical model for the rain effect and show that the well-known atmosphere scattering model (ASM) for the haze effect naturally emerges as its homogeneous continuous limit. Via depth-aware fusion of multi-layer rain streaks according to the camera imaging mechanism, the new model can better capture the sophisticated non-deterministic degradation patterns commonly seen in real rainy images. We also propose a Densely Scale-Connected Attentive Network (DSCAN) that is suitable for both deraining and dehazing tasks. Our design alleviates the bottleneck issue existent in conventiona…
East China Normal University（华东师范大学）；The Chinese University of Hong Kong（香港中文大学）
Single image deraining is important for many high-level computer vision tasks since the rain streaks can severely degrade the visibility of images, thereby affecting the recognition and analysis of the image. Recently, many CNN-based methods have been proposed for rain removal. Although these methods can remove part of the rain streaks, it is difficult for them to adapt to real-world scenarios and restore high-quality rain-free images with clear and accurate structures. To solve this problem, we propose a Structure-Preserving Deraining Network (SPDNet) with RCP guidance. SPDNet directly genera…
Australian National University（澳大利亚国立大学）；Tencent AI Laboratory（腾讯人工智能实验室）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Rain streaks and rain drops are two natural phenomena, which degrade image capture in different ways. Currently, most existing deep deraining networks take them as two distinct problems and individually address one, and thus cannot deal adequately with both simultaneously. To address this, we propose a Dual Attention-in-Attention Model (DAiAM) which includes two DAMs for removing both rain streaks and raindrops. Inside the DAM, there are two attentive maps - each of which attends to the heavy and light rainy regions, respectively, to guide the deraining process differently for applicable regio…
Cross Paradigm Representation and Alignment Transformer for Image Deraining
Nanjing Agricultural University（南京农业大学）；Soochow University（苏州大学）；Xiangtan University（湘潭大学）；Shanghai University（上海大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Westlake University（西湖大学）
Cross-Scenario Deraining Adaptation with Unpaired Data: Superpixel Structural Priors and Multi-Stage Pseudo-Rain Synthesis
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Unpaired Photo-realistic Image Deraining with Energy-informed Diffusion Model
School of Information Engineering, Chang’an University（长安大学信息工程学院）；School of Data Science and Artificial Intelligence, Chang’an University（长安大学数据科学与人工智能学院）
A Hybrid Transformer-Mamba Network for Single Image Deraining
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间科学与技术学院）；Yunnan Normal University（云南师范大学）
Semi-Supervised State-Space Model with Dynamic Stacking Filter for Real-World Video Deraining
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）；MoE Key Laboratory of Information Technology（教育部信息科技重点实验室）；Guangdong Provincial Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Educational Information for Nationalities, Yunnan Normal University（民族教育信息重点实验室，云南师范大学）；School of Mechanical Engineering and Automation, Fuzhou University（福州大学机械工程与自动化学院）
Single-photon emission computed tomography (SPECT) is a widely applied imaging approach for diagnosis of coronary artery diseases. Attenuation maps (u-maps) derived from computed tomography (CT) are utilized for attenuation correction (AC) to improve diagnostic accuracy of cardiac SPECT. However, SPECT and CT are obtained sequentially in clinical practice, which potentially induces misregistration between the two scans. Convolutional neural networks (CNN) are powerful tools for medical image registration. Previous CNN-based methods for cross-modality registration either directly concatenated t…
Nanjing University of Science and Technology（南京理工大学）；NVIDIA（英伟达）；City University of Hong Kong（香港城市大学）；Dalian University of Technology（大连理工大学）；SenseTime Research（商汤科技研究院）；Tencent Youtu Lab（腾讯优图实验室）；UC Merced（加州大学默塞德分校）
In this paper, we propose a general dual convolutional neural network (DualCNN) for low-level vision problems, e.g., super-resolution, edge-preserving filtering, deraining and dehazing. These problems usually involve the estimation of two components of the target signals: structures and details. Motivated by this, our proposed DualCNN consists of two parallel branches, which respectively recovers the structures and details in an end-to-end manner. The recovered structures and details can generate the target signals according to the formation model for each particular application. The DualCNN i…
RVS College of Engineering and Technology（RVS工程技术学院）
In recent years, many research achievements are made in the medical image fusion field. Fusion is basically extraction of best of inputs and conveying it to the output. Medical Image fusion means that several of various modality image information is comprehended together to form one image to express its information. The aim of image fusion is to integrate complementary and redundant information. In this paper, a multimodal image fusion algorithm based on the dual-tree complex wavelet transform (DT-CWT) and adaptive particle swarm optimization (APSO) is proposed. Fusion is achieved through the …
Hanyang University（汉阳大学）；Korea Advanced Institute of Science & Technology (KAIST)（韩国科学技术院（KAIST））
Fusing data from cameras and LiDAR sensors is an essential technique to achieve robust 3D object detection. One key challenge in camera-LiDAR fusion involves mitigating the large domain gap between the two sensors in terms of coordinates and data distribution when fusing their features. In this paper, we propose a novel camera-LiDAR fusion architecture called, 3D Dual-Fusion, which is designed to mitigate the gap between the feature representations of camera and LiDAR data. The proposed method fuses the features of the camera-view and 3D voxel-view domain and models their interactions through …
GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion
College of New Materials and New Energies, Shenzhen Technology University（深圳技术大学新材料与新能源学院）
University of Science and Technology of China（中国科学技术大学）；Huawei Technologies（华为技术有限公司）；Huawei Cloud BU（华为云业务单元）
When smartphone cameras are used to take photos of digital screens, usually moire patterns result, severely degrading photo quality. In this paper, we design a wavelet-based dual-branch network (WDNet) with a spatial attention mechanism for image demoireing. Existing image restoration methods working in the RGB domain have difficulty in distinguishing moire patterns from true scene texture. Unlike these methods, our network removes moire patterns in the wavelet domain to separate the frequencies of moire patterns from the image content. The network combines dense convolution modules and dilate…
Multilevel neural networks with dual-stage feature fusion for human activity recognition
Departamento de Informática, Universidad Carlos III de Madrid（信息学院，卡洛斯三世大学）；Departamento de Teoría de la Señal y Comunicaciones, Universidad Carlos III de Madrid（信号与通信理论系，卡洛斯三世大学）；CEIEC, Universidad Francisco de Vitoria（CEIEC，弗朗西斯科·维多利亚大学）
PRISM: Progressive Rain removal with Integrated State-space Modeling
Faculty of Data Science, City University of Macau, SAR Macao, China（澳门城市大学数据科学学院）；College of Oceanography and Space Informatics, China University of Petroleum (East China), China（中国石油大学（华东）海洋学院）；Zhuhai 4Dage Network Technology, China（珠海4Dage网络技术）
The George Washington University（乔治华盛顿大学）；Children’s National Hospital（国家儿童医院）
Accurate nerve identification is critical during surgical procedures for preventing any damages to nerve tissues. Nerve injuries can lead to long-term detrimental effects for patients as well as financial overburdens. In this study, we develop a deep-learning network framework using the U-Net architecture with a Transformer block based fusion module at the bottleneck to identify nerve tissues from a multi-modal optical imaging system. By leveraging and extracting the feature maps of each modality independently and using each modalities information for cross-modal interactions, we aim to provid…
Recently, image restoration transformers have achieved comparable performance with previous state-of-the-art CNNs. However, how to efficiently leverage such architectures remains an open problem. In this work, we present Dual-former whose critical insight is to combine the powerful global modeling ability of self-attention modules and the local modeling ability of convolutions in an overall architecture. With convolution-based Local Feature Extraction modules equipped in the encoder and the decoder, we only adopt a novel Hybrid Transformer Block in the latent layer to model the long-distance d…
DM-QPMNET: Dual-modality fusion network for cell segmentation in quantitative phase microscopy
College of Computing and Informatics, University of North Carolina at Charlotte, NC, USA（计算与信息学院，北卡罗来纳大学夏洛特分校）；Department of Physics and Optical Science, University of North Carolina at Charlotte, NC, USA（物理与光学科学系，北卡罗来纳大学夏洛特分校）；Center for TAIMing AI, University of North Carolina at Charlotte, NC, USA（TAIMing AI中心，北卡罗来纳大学夏洛特分校）
National University of Defense Technology（国防科技大学）；Tongji University（同济大学）
Wound image segmentation is a critical component for the clinical diagnosis and in-time treatment of wounds. Recently, deep learning has become the mainstream methodology for wound image segmentation. However, the pre-processing of the wound image, such as the illumination correction, is required before the training phase as the performance can be greatly improved. The correction procedure and the training of deep models are independent of each other, which leads to sub-optimal segmentation performance as the fixed illumination correction may not be suitable for all images. To address aforemen…
Shanghai AI Laboratory（上海人工智能实验室）；Uisee Research（驭势科技研究院）；ShanghaiTech University（上海科技大学）；Peking University（北京大学）；University of Pennsylvania（宾夕法尼亚大学）
Point cloud processing is a challenging task due to its sparsity and irregularity. Prior works introduce delicate designs on either local feature aggregator or global geometric architecture, but few combine both advantages. We propose Dual-Scale Point Cloud Recognition with High-frequency Fusion (DSPoint) to extract local-global features by concurrently operating on voxels and points. We reverse the conventional design of applying convolution on voxels and attention to points. Specifically, we disentangle point features through channel dimension for dual-scale processing: one by point-wise con…
School of IoT Engineering, Jiangnan University（江南大学物联网工程学院）；College of Mathematics and Information Science, East China Institute of Technology（东华理工大学数学与信息科学学院）
This paper presents an improved dual channel pulse coupled neural network (IDC-PCNN) model for image fusion. The model can overcome some defects of standard PCNN model. In this fusion scheme, the multiplication rule is replaced by addition rule in the information fusion pool of dual channel PCNN (DC-PCNN) model. Meanwhile the sum of modified Laplacian (SML) measure is adopted, which is better than other focus measures. This method not only inherits the good characteristics of the standard PCNN model but also enhances the computing efficiency and fusion quality. The performance of the proposed …
MAUGIF: Mechanism-Aware Unsupervised General Image Fusion via Dual Cross-Image Autoencoders
School of Mathematics, Hunan University（湖南大学数学学院）；College of Information Engineering, Zhejiang University of Technology（浙江工业大学信息工程学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）
