Robust semantic scene segmentation for automotive applications is a challenging problem in two key aspects: (1) labelling every individual scene pixel and (2) performing this task under unstable weather and illumination changes (e.g., foggy weather), which results in poor outdoor scene visibility. Such visibility limitations lead to non-optimal performance of generalised deep convolutional neural network-based semantic scene segmentation. In this paper, we propose an efficient end-to-end automotive semantic scene understanding approach that is robust to foggy weather conditions. As an end-to-end pipeline, our proposed approach provides: (1) the transformation of imagery from foggy to clear weather conditions using a domain transfer approach (correcting for poor visibility) and (2) semantically segmenting the scene using a competitive encoder-decoder architecture with low computational complexity (enabling real-time performance). Our approach incorporates RGB colour, depth and luminance images via distinct encoders with dense connectivity and features fusion to effectively exploit information from different inputs, which contributes to an optimal feature representation within the overall model. Using this architectural formulation with dense skip connections, our model achieves comparable performance to contemporary approaches at a fraction of the overall model complexity.
Joint scene understanding and segmentation for automotive applications is a challenging problem in two key aspects:- (1) classifying every pixel in the entire scene and (2) performing this task under unstable weather and illumination changes (e.g. foggy weather), which results in poor outdoor scene visibility. This poor outdoor scene visibility leads to a non-optimal performance of deep convolutional neural network-based scene understanding and segmentation. In this paper, we propose an efficient end-to-end contemporary automotive semantic scene understanding approach under foggy weather condi…
Huazhong University of Science and Technology（华中科技大学）；Huawei International Co. Ltd.（华为国际有限公司）
Optical flow has achieved great success under clean scenes, but suffers from restricted performance under foggy scenes. To bridge the clean-to-foggy domain gap, the existing methods typically adopt the domain adaptation to transfer the motion knowledge from clean to synthetic foggy domain. However, these methods unexpectedly neglect the synthetic-to-real domain gap, and thus are erroneous when applied to real-world scenes. To handle the practical optical flow under real foggy scenes, in this work, we propose a novel unsupervised cumulative domain adaptation optical flow (UCDA-Flow) framework: …
Automotive scene understanding under adverse weather conditions raises a realistic and challenging problem attributable to poor outdoor scene visibility (e.g. foggy weather). However, because most contemporary scene understanding approaches are applied under ideal-weather conditions, such approaches may not provide genuinely optimal performance when compared to established a priori insights on extreme-weather understanding. In this paper, we propose a complex but competitive multi-task learning approach capable of performing in real-time semantic scene understanding and monocular depth estimat…
National University of Singapore（新加坡国立大学）；Huawei International Pte Ltd（华为国际私人有限公司）；Yale-NUS College（耶鲁-新加坡国立大学学院）
Most existing domain adaptation (DA) methods align the features based on the domain feature distributions and ignore aspects related to fog, background and target objects, rendering suboptimal performance. In our DA framework, we retain the depth and background information during the domain feature alignment. A consistency loss between the generated depth and fog transmission map is introduced to strengthen the retention of the depth information in the aligned features. To address false object features potentially generated during the DA process, we propose an encoder-decoder framework to reco…
Wuhan University（武汉大学）；UTokyo（东京大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；NTHU（台湾清华大学）
Although considerable progress has been made in semantic scene understanding under clear weather, it is still a tough problem under adverse weather conditions, such as dense fog, due to the uncertainty caused by imperfect observations. Besides, difficulties in collecting and labeling foggy images hinder the progress of this field. Considering the success in semantic scene understanding under clear weather, we think it is reasonable to transfer knowledge learned from clear images to the foggy domain. As such, the problem becomes to bridge the domain gap between clear images and foggy images. Un…
National Institute of Informatics（国立信息学研究所）；Wuhan University（武汉大学）；National Tsing Hua University（国立清华大学）
Understanding foggy image sequence in the driving scenes is critical for autonomous driving, but it remains a challenging task due to the difficulty in collecting and annotating real-world images of adverse weather. Recently, the self-training strategy has been considered a powerful solution for unsupervised domain adaptation, which iteratively adapts the model from the source domain to the target domain by generating target pseudo labels and re-training the model. However, the selection of confident pseudo labels inevitably suffers from the conflict between sparsity and accuracy, both of whic…
Fourier Domain Adaptation for Traffic Light Detection in Adverse Weather
Manipal Institute of Technology（曼尼帕尔理工学院）；Manipal Academy of Higher Education（曼尼帕尔高等教育学院）；Wells Fargo International Solutions Private Limited（富国国际解决方案私人有限公司）
FogGuard: guarding YOLO against fog using perceptual loss
School of Computing and Information Science, University of Maine（计算与信息科学学院，缅因大学）；Department of Electrical and Computer Engineering, University of Maine（电气与计算机工程系，缅因大学）
Object detection is an essential technique for autonomous driving. The performance of an object detector significantly degrades if the weather of the training images is different from that of test images. Domain adaptation can be used to address the domain shift problem so as to improve the robustness of an object detector. However, most existing domain adaptation methods either handle single target domain or require domain labels. We propose a novel unsupervised domain classification method which can be used to generalize single-target domain adaptation methods to multi-target domains, and de…
Defect detection is a basic and essential task in automatic parts production, especially for automotive engine precision parts. In this paper, we propose a new idea to construct a deep convolutional network combining related knowledge of feature processing and the representation ability of deep learning. Our algorithm consists of a pixel-wise segmentation Deep Neural Network (DNN) and a feature refining network. The fully convolutional DNN is presented to learn basic features of parts defects. After that, several typical traditional methods which are used to refine the segmentation results are…
This work addresses the problem of semantic scene understanding under dense fog. Although considerable progress has been made in semantic scene understanding, it is mainly related to clear-weather scenes. Extending recognition methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both synthetic and real foggy data. In addition, we present three other main sta…
This work addresses the problem of semantic scene understanding under fog. Although marked progress has been made in semantic scene understanding, it is mainly concentrated on clear-weather scenes. Extending semantic segmentation methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both labeled synthetic foggy data and unlabeled real foggy data. The method i…
This paper presents FogAdapt, a novel approach for domain adaptation of semantic segmentation for dense foggy scenes. Although significant research has been directed to reduce the domain shift in semantic segmentation, adaptation to scenes with adverse weather conditions remains an open question. Large variations in the visibility of the scene due to weather conditions, such as fog, smog, and haze, exacerbate the domain shift, thus making unsupervised adaptation in such scenarios challenging. We propose a self-entropy and multi-scale information augmented self-supervised domain adaptation meth…
Cleveland State University（克利夫兰州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Wuhan University（武汉大学）
Most object detection methods for autonomous driving usually assume a consistent feature distribution between training and testing data, which is not always the case when weathers differ significantly. The object detection model trained under clear weather might not be effective enough in foggy weather because of the domain gap. This paper proposes a novel domain adaptive object detection framework for autonomous driving under foggy weather. Our method leverages both image-level and object-level adaptation to diminish the domain discrepancy in image style and object appearance. To further enha…
Advanced collision avoidance and driver hand-off systems can benefit from the ability to accurately predict, in real time, the probability a vehicle will be involved in a collision within an intermediate horizon of 10 to 20 seconds. The rarity of collisions in real-world data poses a significant challenge to developing this capability because, as we demonstrate empirically, intermediate-horizon risk prediction depends heavily on high-dimensional driver behavioral features. As a result, a large amount of data is required to fit an effective predictive model. In this paper, we assess whether sim…
Shanghai Jiao Tong University（上海交通大学）；East China Normal University（华东师范大学）
Domain adaptive object detection (DAOD) aims to alleviate transfer performance degradation caused by the cross-domain discrepancy. However, most existing DAOD methods are dominated by outdated and computationally intensive two-stage Faster R-CNN, which is not the first choice for industrial applications. In this paper, we propose a novel semi-supervised domain adaptive YOLO (SSDA-YOLO) based method to improve cross-domain detection performance by integrating the compact one-stage stronger detector YOLOv5 with domain adaptation. Specifically, we adapt the knowledge distillation framework with t…
Georgia Institute of Technology（佐治亚理工学院）；Twentieth Century Fox（二十世纪福克斯公司）；Starfleet Academy（星际舰队学院）；Tyrell Inc.（泰瑞尔公司）
Systems for the automatic recognition and detection of automotive parts are crucial in several emerging research areas in the development of intelligent vehicles. They enable, for example, the detection and modelling of interactions between human and the vehicle. In this paper, we quantitatively and qualitatively explore the efficacy of deep learning architectures for the classification and localisation of 29 interior and exterior vehicle regions on three novel datasets. Furthermore, we experiment with joint and transfer learning approaches across datasets and point out potential applications …
UMDATrack: Unified Multi-Domain Adaptive Tracking Under Adverse Weather Conditions
Beijing University of Posts and Telecommunications（北京邮电大学）；Sun Yat-sen University（中山大学）；University of Chinese Academy of Sciences（中国科学院大学）；MoE Key Laboratory of Information Technology（教育部信息科学技术重点实验室）；Guangdong Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）
Towards Railway Domain Adaptation for LiDAR-based 3D Detection: Road-to-Rail and Sim-to-Real via SynDRA-BBox
Mobility Technologies（移动技术）；Department of Excellence in Robotics & AI, SETLabs Research GmbH（机器人与人工智能卓越部门，SETLabs Research GmbH）；Scuola Superiore Sant’Anna（圣安娜高等学院）
University of Science and Technology of China（中国科学技术大学）；JD AI Research（京东人工智能研究院）；City University of Hong Kong（香港城市大学）；Peking University（北京大学）
Rendering synthetic data (e.g., 3D CAD-rendered images) to generate annotations for learning deep models in vision tasks has attracted increasing attention in recent years. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. To address this issue, recent progress in cross-domain recognition has featured the Mean Teacher, which directly simulates unsupervised domain adaptation as semi-supervised learning. The domain gap is thus naturally bridged with consistency regularization in a teacher-student scheme. In th…
Despite growing interest in object detection, very few works address the extremely practical problem of cross-domain robustness especially for automative applications. In order to prevent drops in performance due to domain shift, we introduce an unsupervised domain adaptation method built on the foundation of faster-RCNN with two domain adaptation components addressing the shift at the instance and image levels respectively and apply a consistency regularization between them. We also introduce a family of adaptation layers that leverage the squeeze excitation mechanism called SE Adaptors to im…
Univrses（Univrses公司）；Kudan（Kudan公司）；Technical University of Munich（慕尼黑工业大学）；University of Bologna（博洛尼亚大学）
Unsupervised Domain Adaptation (UDA) aims at reducing the domain gap between training and testing data and is, in most cases, carried out in offline manner. However, domain changes may occur continuously and unpredictably during deployment (e.g. sudden weather changes). In such conditions, deep neural networks witness dramatic drops in accuracy and offline adaptation may not be enough to contrast it. In this paper, we tackle Online Domain Adaptation (OnDA) for semantic segmentation. We design a pipeline that is robust to continuous domain shifts, either gradual or sudden, and we evaluate it in…
Beijing Institute of Technology（北京理工大学）；Tsinghua University（清华大学）；Inceptio Technology（嬴彻科技）
Generalizing models trained on normal visual conditions to target domains under adverse conditions is demanding in the practical systems. One prevalent solution is to bridge the domain gap between clear- and adverse-condition images to make satisfactory prediction on the target. However, previous methods often reckon on additional reference images of the same scenes taken from normal conditions, which are quite tough to collect in reality. Furthermore, most of them mainly focus on individual adverse condition such as nighttime or foggy, weakening the model versatility when encountering other a…
ETH Zurich（苏黎世联邦理工学院）；MPI for Informatics（马克斯·普朗克信息学研究所）；KU Leuven（鲁汶大学）
Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. It can save the cost of manually labeling data in real-world applications such as robot vision and autonomous driving. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. However, such an assumption does not always hold in practice owing to the collection difficulty and the scarcity of the data. Thus, we aim to relieve this need on a large number of real …
UniDA3D: A Unified Domain-Adaptive Framework for Multi-View 3D Object Detection
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全全国重点实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）
Adverse weather conditions such as haze and rain corrupt the quality of captured images, which cause detection networks trained on clean images to perform poorly on these images. To address this issue, we propose an unsupervised prior-based domain adversarial object detection framework for adapting the detectors to hazy and rainy conditions. In particular, we use weather-specific prior knowledge obtained using the principles of image formation to define a novel prior-adversarial loss. The prior-adversarial loss used to train the adaptation process aims to reduce the weather-specific informatio…
This work addresses the unsupervised adaptation of an existing object detector to a new target domain. We assume that a large number of unlabeled videos from this domain are readily available. We automatically obtain labels on the target data by using high-confidence detections from the existing detector, augmented with hard (misclassified) examples acquired by exploiting temporal cues using a tracker. These automatically-obtained labels are then used for re-training the original model. A modified knowledge distillation loss is proposed, and we investigate several ways of assigning soft-labels…
One of the key problems in computer vision is adaptation: models are too rigid to follow the variability of the inputs. The canonical computation that explains adaptation in sensory neuroscience is divisive normalization, and it has appealing effects on image manifolds. In this work we show that including divisive normalization in current deep networks makes them more invariant to non-informative changes in the images. In particular, we focus on U-Net architectures for image segmentation. Experiments show that the inclusion of divisive normalization in the U-Net architecture leads to better se…
Towards Unbiased Source-Free Object Detection via Vision Foundation Models
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, 100191, China（复杂与关键软件环境国家重点实验室，北京航空航天大学，北京，100191，中国）；School of Computer Science and Engineering, Beihang University, Beijing, 100191, China（计算机科学与工程学院，北京航空航天大学，北京，100191，中国）；School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, China（计算机科学与信息工程学院，合肥工业大学，合肥，230601，中国）
Valeo India（法雷奥印度公司）；National University of Ireland, Galway（爱尔兰国立高威大学）；Indian Statistical Institute（印度统计研究所）；Valeo Vision Systems（法雷奥视觉系统公司）；University of Limerick（利默里克大学）
Most of the existing works on pedestrian pose estimation do not consider estimating the pose of an occluded pedestrian, as the annotations of the occluded parts are not available in relevant automotive datasets. For example, CityPersons, a well-known dataset for pedestrian detection in automotive scenes does not provide pose annotations, whereas MS-COCO, a non-automotive dataset, contains human pose estimation. In this work, we propose a multi-task framework to extract pedestrian features through detection and instance segmentation tasks performed separately on these two distributions. Thereaf…
Real-Time Scene-Adaptive Tone Mapping for High-Dynamic Range Object Detection
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Tianmushan Laboratory, Beihang University（北京航空航天大学天目山实验室）；School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息工程大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Point Spread Technology（点扩散科技）
Valeo Troy（法雷奥特洛伊）；Valeo Vision Systems（法雷奥视觉系统）；University of Michigan-Dearborn（密歇根大学迪尔伯恩分校）
Decision making in automated driving is highly specific to the environment and thus semantic segmentation plays a key role in recognizing the objects in the environment around the car. Pixel level classification once considered a challenging task which is now becoming mature to be productized in a car. However, semantic annotation is time consuming and quite expensive. Synthetic datasets with domain adaptation techniques have been used to alleviate the lack of large annotated datasets. In this work, we explore an alternate approach of leveraging the annotations of other tasks to improve semant…
