Adverse weather image translation belongs to the unsupervised image-to-image (I2I) translation task which aims to transfer adverse condition domain (eg, rainy night) to standard domain (eg, day). It is a challenging task because images from adverse domains have some artifacts and insufficient information. Recently, many studies employing Generative Adversarial Networks (GANs) have achieved notable success in I2I translation but there are still limitations in applying them to adverse weather enhancement. Symmetric architecture based on bidirectional cycle-consistency loss is adopted as a standard framework for unsupervised domain transfer methods. However, it can lead to inferior translation result if the two domains have imbalanced information. To address this issue, we propose a novel GAN model, i.e., AU-GAN, which has an asymmetric architecture for adverse domain translation. We insert a proposed feature transfer network (${T}$-net) in only a normal domain generator (i.e., rainy night-> day) to enhance encoded features of the adverse domain image. In addition, we introduce asymmetric feature matching for disentanglement of encoded features. Finally, we propose uncertainty-aware cycle-consistency loss to address the regional uncertainty of a cyclic reconstructed image. We demonstrate the effectiveness of our method by qualitative and quantitative comparisons with state-of-the-art models. Codes are available at https://github.com/jgkwak95/AU-GAN.
We presented a method for improving computer vision tasks on images affected by adverse weather conditions, including distortions caused by adherent raindrops. Overcoming the challenge of applying computer vision to images affected by adverse weather conditions is essential for autonomous vehicles utilizing RGB cameras. For this purpose, we trained an appropriate generative adversarial network and showed that it was effective at removing the effect of the distortions, in the context of image reconstruction and computer vision tasks. We showed that object recognition, a vital task for autonomou…
SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal
School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；College of Engineering, Huaqiao University（华侨大学工学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间安全学院）
Northwestern Polytechnical University（西北工业大学）；School of Artificial Intelligence, Optics and Electronics (iOPEN)（人工智能、光学与电子学院（iOPEN））；School of Computer Science（计算机科学学院）
In this paper, a new task is proposed, namely, weather translation, which refers to transferring weather conditions of the image from one category to another. It is important for photographic style transfer. Although lots of approaches have been proposed in traditional image translation tasks, few of them can handle the multi-category weather translation task, since weather conditions have rich categories and highly complex semantic structures. To address this problem, we develop a multi-domain weather translation approach based on generative adversarial networks (GAN), denoted as Weather GAN,…
Convolutional neural network (CNN) have proven its success for semantic segmentation, which is a core task of emerging industrial applications such as autonomous driving. However, most progress in semantic segmentation of urban scenes is reported on standard scenarios, i.e., daytime scenes with favorable illumination conditions. In practical applications, the outdoor weather and illumination are changeable, e.g., cloudy and nighttime, which results in a significant drop of semantic segmentation accuracy of CNN only trained with daytime data. In this paper, we propose a novel generative adversa…
ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions
Faculty of Engineering and Information Technology, University of Technology Sydney（工程与信息技术学院，技术悉尼大学）；PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（智能感知与高维信息系统教育部重点实验室，计算机科学与工程学院，南京理工大学）；Key Laboratory of Artificial Intelligence, Ministry of Education（人工智能教育部重点实验室）
Technical University of Munich（慕尼黑工业大学）；Artisense；Audi Electronics Venture（奥迪电子创投）
Even though end-to-end supervised learning has shown promising results for sensorimotor control of self-driving cars, its performance is greatly affected by the weather conditions under which it was trained, showing poor generalization to unseen conditions. In this paper, we show how knowledge can be transferred using semantic maps to new weather conditions without the need to obtain new ground truth data. To this end, we propose to divide the task of vehicle control into two independent modules: a control module which is only trained on one weather condition for which labeled steering data is…
University of Warwick（华威大学）；TUM（慕尼黑工业大学）；LSE（伦敦政治经济学院）；DLR（德国航空航天中心）
Deep generative models are increasingly used to gain insights in the geospatial data domain, e.g., for climate data. However, most existing approaches work with temporal snapshots or assume 1D time-series; few are able to capture spatio-temporal processes simultaneously. Beyond this, Earth-systems data often exhibit highly irregular and complex patterns, for example caused by extreme weather events. Because of climate change, these phenomena are only increasing in frequency. Here, we proposed a novel GAN-based approach for generating spatio-temporal weather patterns conditioned on detected ext…
Lancaster University（兰卡斯特大学）；Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）
Recent semantic segmentation models perform well under standard weather conditions and sufficient illumination but struggle with adverse weather conditions and nighttime. Collecting and annotating training data under these conditions is expensive, time-consuming, error-prone, and not always practical. Usually, synthetic data is used as a feasible data source to increase the amount of training data. However, just directly using synthetic data may actually harm the model's performance under normal weather conditions while getting only small gains in adverse situations. Therefore, we present a no…
Despite the recent advances of deep neural networks, object detection for adverse weather remains challenging due to the poor perception of some sensors in adverse weather. Instead of relying on one single sensor, multimodal fusion has been one promising approach to provide redundant detection information based on multiple sensors. However, most existing multimodal fusion approaches are ineffective in adjusting the focus of different sensors under varying detection environments in dynamic adverse weather conditions. Moreover, it is critical to simultaneously observe local and global informatio…
Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；University of Cambridge（剑桥大学）；Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）；Tianjin University（天津大学）；Hong Kong Metropolitan University（香港都会大学）
Reliable point cloud data is essential for perception tasks \textit{e.g.} in robotics and autonomous driving applications. Adverse weather causes a specific type of noise to light detection and ranging (LiDAR) sensor data, which degrades the quality of the point clouds significantly. To address this issue, this letter presents a novel point cloud adverse weather denoising deep learning algorithm (4DenoiseNet). Our algorithm takes advantage of the time dimension unlike deep learning adverse weather denoising methods in the literature. It performs about 10\% better in terms of intersection over …
Uncertainty-Guided Adverse Weather Restoration via Gated Transformer Network
School of Automation, Southeast University（东南大学自动化学院）；Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich（慕尼黑工业大学机器人、人工智能与实时系统讲席教授组）
Unifying Physically-Informed Weather Priors in A Single Model for Image Restoration Across Multiple Adverse Weather Conditions
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学（深圳）计算机科学与工程系）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室）；ROAS Thrust, the Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China and The Hong Kong University of Science and Technology, Department of Electronic and Computer Engineering, Hong Kong SAR, China（香港科学与技术大学（广州）ROAS方向及电子与计算机工程系，香港特别行政区）
3D Semantic Segmentation in the Wild: Learning Generalized Models for Adverse-Condition Point Clouds
Nanyang Technological University（南洋理工大学）；Waseda University（早稻田大学）；Technical University of Denmark（丹麦技术大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Ottawa（渥太华大学）；Carnegie Mellon University（卡内基梅隆大学）
Segmenting each moving object instance in a scene is essential for many applications. But like many other computer vision tasks, this task performs well in optimal weather, but then adverse weather tends to fail. To be robust in weather conditions, the usual way is to train network in data of given weather pattern or to fuse multiple sensors. We focus on a new possibility, that is, to improve its resilience to weather interference through the network's structural design. First, we propose a novel FPN structure called RiWFPN with a progressive top-down interaction and attention refinement modul…
Robust Single Object Tracking in LiDAR Point Clouds under Adverse Weather Conditions
School of Mathematical Sciences, Dalian University of Technology（大连理工大学数学科学学院）；School of Economics and Management, China University of Mining and Technology（中国矿业大学经济管理学院）；DUT-BSU Joint Institute, Dalian University of Technology（大连理工大学-白俄罗斯国立大学联合学院）
UniDet-D: A Unified Dynamic Spectral Attention Model for Object Detection under Adverse Weathers
School of Interdisciplinary Science, Beijing Institute of Technology（交叉学科学院，北京理工大学）；North China University of Technology（华北理工大学）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology（空间智能信息处理国家重点实验室，北京理工大学）；School of Computer Science and Engineering, Nanyang Technological University（计算机科学与工程学院，南洋理工大学）
National Key Laboratory of Science and Technology on Multispectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院多谱信息处理技术国家级重点实验室）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
Optical flow estimation has made great progress, but usually suffers from degradation under adverse weather. Although semi/full-supervised methods have made good attempts, the domain shift between the synthetic and real adverse weather images would deteriorate their performance. To alleviate this issue, our start point is to unsupervisedly transfer the knowledge from source clean domain to target degraded domain. Our key insight is that adverse weather does not change the intrinsic optical flow of the scene, but causes a significant difference for the warp error between clean and degraded imag…
Lancaster University（兰卡斯特大学）；Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）
Video stabilization plays a central role to improve videos quality. However, despite the substantial progress made by these methods, they were, mainly, tested under standard weather and lighting conditions, and may perform poorly under adverse conditions. In this paper, we propose a synthetic-aware adverse weather robust algorithm for video stabilization that does not require real data and can be trained only on synthetic data. We also present Silver, a novel rendering engine to generate the required training data with an automatic ground-truth extraction procedure. Our approach uses our speci…
Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the a…
Though deep learning-based object detection methods have achieved promising results on the conventional datasets, it is still challenging to locate objects from the low-quality images captured in adverse weather conditions. The existing methods either have difficulties in balancing the tasks of image enhancement and object detection, or often ignore the latent information beneficial for detection. To alleviate this problem, we propose a novel Image-Adaptive YOLO (IA-YOLO) framework, where each image can be adaptively enhanced for better detection performance. Specifically, a differentiable ima…
Adverse weather conditions are very challenging for autonomous driving because most of the state-of-the-art sensors stop working reliably under these conditions. In order to develop robust sensors and algorithms, tests with current sensors in defined weather conditions are crucial for determining the impact of bad weather for each sensor. This work describes a testing and evaluation methodology that helps to benchmark novel sensor technologies and compare them to state-of-the-art sensors. As an example, gated imaging is compared to standard imaging under foggy conditions. It is shown that gate…
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院多光谱信息智能处理技术国家重点实验室）；Huawei International Co. Ltd.（华为国际有限公司）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Electrical Engineering and Computer Science, Peking University（北京大学电气与计算机工程学院）
Daimler AG（戴姆勒股份公司）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Lidar sensors are frequently used in environment perception for autonomous vehicles and mobile robotics to complement camera, radar, and ultrasonic sensors. Adverse weather conditions are significantly impacting the performance of lidar-based scene understanding by causing undesired measurement points that in turn effect missing detections and false positives. In heavy rain or dense fog, water drops could be misinterpreted as objects in front of the vehicle which brings a mobile robot to a full stop. In this paper, we present the first CNN-based approach to understand and filter out such adver…
RoSe: Robust Self-supervised Stereo Matching under Adverse Weather Conditions
Department of Computer Science, City University of Hong Kong（城市大学计算机科学系）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所（CASIA））
Oxford Brookes University（牛津布鲁克斯大学）；University of Oxford（牛津大学）
In an autonomous driving system, perception - identification of features and objects from the environment - is crucial. In autonomous racing, high speeds and small margins demand rapid and accurate detection systems. During the race, the weather can change abruptly, causing significant degradation in perception, resulting in ineffective manoeuvres. In order to improve detection in adverse weather, deep-learning-based models typically require extensive datasets captured in such conditions - the collection of which is a tedious, laborious, and costly process. However, recent developments in Cycl…
