Image restoration under adverse weather conditions has been of significant interest for various computer vision applications. Recent successful methods rely on the current progress in deep neural network architectural designs (e.g., with vision transformers). Motivated by the recent progress achieved with state-of-the-art conditional generative models, we present a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models. Our patch-based diffusion modeling approach enables size-agnostic image restoration by using a guided denoising process with smoothed noise estimates across overlapping patches during inference. We empirically evaluate our model on benchmark datasets for image desnowing, combined deraining and dehazing, and raindrop removal. We demonstrate our approach to achieve state-of-the-art performances on both weather-specific and multi-weather image restoration, and experimentally show strong generalization to real-world test images.
We presented a method for improving computer vision tasks on images affected by adverse weather conditions, including distortions caused by adherent raindrops. Overcoming the challenge of applying computer vision to images affected by adverse weather conditions is essential for autonomous vehicles utilizing RGB cameras. For this purpose, we trained an appropriate generative adversarial network and showed that it was effective at removing the effect of the distortions, in the context of image reconstruction and computer vision tasks. We showed that object recognition, a vital task for autonomou…
SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal
School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；College of Engineering, Huaqiao University（华侨大学工学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）
Adverse weather image translation belongs to the unsupervised image-to-image (I2I) translation task which aims to transfer adverse condition domain (eg, rainy night) to standard domain (eg, day). It is a challenging task because images from adverse domains have some artifacts and insufficient information. Recently, many studies employing Generative Adversarial Networks (GANs) have achieved notable success in I2I translation but there are still limitations in applying them to adverse weather enhancement. Symmetric architecture based on bidirectional cycle-consistency loss is adopted as a standa…
ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions
Faculty of Engineering and Information Technology, University of Technology Sydney（工程与信息技术学院，技术悉尼大学）；PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（智能感知与高维信息系统教育部重点实验室，计算机科学与工程学院，南京理工大学）；Key Laboratory of Artificial Intelligence, Ministry of Education（人工智能教育部重点实验室）
Lancaster University（兰卡斯特大学）；Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）
Recent semantic segmentation models perform well under standard weather conditions and sufficient illumination but struggle with adverse weather conditions and nighttime. Collecting and annotating training data under these conditions is expensive, time-consuming, error-prone, and not always practical. Usually, synthetic data is used as a feasible data source to increase the amount of training data. However, just directly using synthetic data may actually harm the model's performance under normal weather conditions while getting only small gains in adverse situations. Therefore, we present a no…
Despite the recent advances of deep neural networks, object detection for adverse weather remains challenging due to the poor perception of some sensors in adverse weather. Instead of relying on one single sensor, multimodal fusion has been one promising approach to provide redundant detection information based on multiple sensors. However, most existing multimodal fusion approaches are ineffective in adjusting the focus of different sensors under varying detection environments in dynamic adverse weather conditions. Moreover, it is critical to simultaneously observe local and global informatio…
Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；University of Cambridge（剑桥大学）；Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）；Tianjin University（天津大学）；Hong Kong Metropolitan University（香港都会大学）
Reliable point cloud data is essential for perception tasks \textit{e.g.} in robotics and autonomous driving applications. Adverse weather causes a specific type of noise to light detection and ranging (LiDAR) sensor data, which degrades the quality of the point clouds significantly. To address this issue, this letter presents a novel point cloud adverse weather denoising deep learning algorithm (4DenoiseNet). Our algorithm takes advantage of the time dimension unlike deep learning adverse weather denoising methods in the literature. It performs about 10\% better in terms of intersection over …
Uncertainty-Guided Adverse Weather Restoration via Gated Transformer Network
School of Automation, Southeast University（东南大学自动化学院）；Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich（慕尼黑工业大学机器人、人工智能与实时系统讲席教授组）
Unifying Physically-Informed Weather Priors in A Single Model for Image Restoration Across Multiple Adverse Weather Conditions
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学（深圳）计算机科学与工程系）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室）；ROAS Thrust, the Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China and The Hong Kong University of Science and Technology, Department of Electronic and Computer Engineering, Hong Kong SAR, China（香港科学与技术大学（广州）ROAS方向及电子与计算机工程系，香港特别行政区）
3D Semantic Segmentation in the Wild: Learning Generalized Models for Adverse-Condition Point Clouds
Nanyang Technological University（南洋理工大学）；Waseda University（早稻田大学）；Technical University of Denmark（丹麦技术大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Ottawa（渥太华大学）；Carnegie Mellon University（卡内基梅隆大学）
Segmenting each moving object instance in a scene is essential for many applications. But like many other computer vision tasks, this task performs well in optimal weather, but then adverse weather tends to fail. To be robust in weather conditions, the usual way is to train network in data of given weather pattern or to fuse multiple sensors. We focus on a new possibility, that is, to improve its resilience to weather interference through the network's structural design. First, we propose a novel FPN structure called RiWFPN with a progressive top-down interaction and attention refinement modul…
Robust Single Object Tracking in LiDAR Point Clouds under Adverse Weather Conditions
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Economics and Management, China University of Mining and Technology（中国矿业大学经济管理学院）；DUT-BSU Joint Institute, Dalian University of Technology（大连理工大学-白俄罗斯国立大学联合学院）
UniDet-D: A Unified Dynamic Spectral Attention Model for Object Detection under Adverse Weathers
School of Interdisciplinary Science, Beijing Institute of Technology（交叉学科学院，北京理工大学）；North China University of Technology（华北理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology（空间智能信息处理国家重点实验室，北京理工大学）；School of Computer Science and Engineering, Nanyang Technological University（计算机科学与工程学院，南洋理工大学）
A robust and reliable semantic segmentation in adverse weather conditions is very important for autonomous cars, but most state-of-the-art approaches only achieve high accuracy rates in optimal weather conditions. The reason is that they are only optimized for good weather conditions and given noise models. However, most of them fail, if data with unknown disturbances occur, and their performance decrease enormously. One possibility to still obtain reliable results is to observe the environment with different sensor types, such as camera and lidar, and to fuse the sensor data by means of neura…
Computer vision tasks such as semantic segmentation perform very well in good weather conditions, but if the weather turns bad, they have problems to achieve this performance in these conditions. One possibility to obtain more robust and reliable results in adverse weather conditions is to use video-segmentation approaches instead of commonly used single-image segmentation methods. Video-segmentation approaches capture temporal information of the previous video-frames in addition to current image information, and hence, they are more robust against disturbances, especially if they occur in onl…
ETH Zürich（苏黎世联邦理工学院）；MPI for Informatics（马克斯·普朗克信息学研究所）；KU Leuven（鲁汶大学）
This work addresses the challenging task of LiDAR-based 3D object detection in foggy weather. Collecting and annotating data in such a scenario is very time, labor and cost intensive. In this paper, we tackle this problem by simulating physically accurate fog into clear-weather scenes, so that the abundant existing real datasets captured in clear weather can be repurposed for our task. Our contributions are twofold: 1) We develop a physically valid fog simulation method that is applicable to any LiDAR dataset. This unleashes the acquisition of large-scale foggy training data at no extra cost. …
National Key Laboratory of Science and Technology on Multispectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院多谱信息处理技术国家级重点实验室）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
Optical flow estimation has made great progress, but usually suffers from degradation under adverse weather. Although semi/full-supervised methods have made good attempts, the domain shift between the synthetic and real adverse weather images would deteriorate their performance. To alleviate this issue, our start point is to unsupervisedly transfer the knowledge from source clean domain to target degraded domain. Our key insight is that adverse weather does not change the intrinsic optical flow of the scene, but causes a significant difference for the warp error between clean and degraded imag…
Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the a…
Though deep learning-based object detection methods have achieved promising results on the conventional datasets, it is still challenging to locate objects from the low-quality images captured in adverse weather conditions. The existing methods either have difficulties in balancing the tasks of image enhancement and object detection, or often ignore the latent information beneficial for detection. To alleviate this problem, we propose a novel Image-Adaptive YOLO (IA-YOLO) framework, where each image can be adaptively enhanced for better detection performance. Specifically, a differentiable ima…
Adverse weather conditions are very challenging for autonomous driving because most of the state-of-the-art sensors stop working reliably under these conditions. In order to develop robust sensors and algorithms, tests with current sensors in defined weather conditions are crucial for determining the impact of bad weather for each sensor. This work describes a testing and evaluation methodology that helps to benchmark novel sensor technologies and compare them to state-of-the-art sensors. As an example, gated imaging is compared to standard imaging under foggy conditions. It is shown that gate…
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院多光谱信息智能处理技术国家重点实验室）；Huawei International Co. Ltd.（华为国际有限公司）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Electrical Engineering and Computer Science, Peking University（北京大学电气与计算机工程学院）
Normandie Univ（诺曼底大学）；INSA Rouen（鲁昂国立应用科学学院）；UNIROUEN（鲁昂大学）；UNIHAVRE（勒阿弗尔大学）；LITIS（LITIS实验室）；University of Burgundy（勃艮第大学）；UBFC（勃艮第-弗朗什孔泰大学）；ImViA（ImViA实验室）
Object detection in road scenes is necessary to develop both autonomous vehicles and driving assistance systems. Even if deep neural networks for recognition task have shown great performances using conventional images, they fail to detect objects in road scenes in complex acquisition situations. In contrast, polarization images, characterizing the light wave, can robustly describe important physical properties of the object even under poor illumination or strong reflections. This paper shows how non-conventional polarimetric imaging modality overcomes the classical methods for object detectio…
Daimler AG（戴姆勒股份公司）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Lidar sensors are frequently used in environment perception for autonomous vehicles and mobile robotics to complement camera, radar, and ultrasonic sensors. Adverse weather conditions are significantly impacting the performance of lidar-based scene understanding by causing undesired measurement points that in turn effect missing detections and false positives. In heavy rain or dense fog, water drops could be misinterpreted as objects in front of the vehicle which brings a mobile robot to a full stop. In this paper, we present the first CNN-based approach to understand and filter out such adver…
RoSe: Robust Self-supervised Stereo Matching under Adverse Weather Conditions
Department of Computer Science, City University of Hong Kong（城市大学计算机科学系）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所（CASIA））
Restoring Images in Adverse Weather Conditions via Histogram Transformer
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）；Guangdong Provincial Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Wechat Business Group, Tencent（腾讯微信事业群）
