The high performance of RGB-D based road segmentation methods contrasts with their rare application in commercial autonomous driving, which is owing to two reasons: 1) the prior methods cannot achieve high inference speed and high accuracy in both ways; 2) the different properties of RGB and depth data are not well-exploited, limiting the reliability of predicted road. In this paper, based on the evidence theory, an uncertainty-aware symmetric network (USNet) is proposed to achieve a trade-off between speed and accuracy by fully fusing RGB and depth data. Firstly, cross-modal feature fusion operations, which are indispensable in the prior RGB-D based methods, are abandoned. We instead separately adopt two light-weight subnetworks to learn road representations from RGB and depth inputs. The light-weight structure guarantees the real-time inference of our method. Moreover, a multiscale evidence collection (MEC) module is designed to collect evidence in multiple scales for each modality, which provides sufficient evidence for pixel class determination. Finally, in uncertainty-aware fusion (UAF) module, the uncertainty of each modality is perceived to guide the fusion of the two subnetworks. Experimental results demonstrate that our method achieves a state-of-the-art accuracy with real-time inference speed of 43+ FPS. The source code is available at https://github.com/morancyc/USNet.
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑科学智能科技研究院）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
Politecnico di Torino（都灵理工大学）；University of Tübingen（图宾根大学）；Technical University of Munich（慕尼黑技术大学）；Helmholtz Munich（海德堡-慕尼黑 Helmholtz 中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Munich Data Science Institute（慕尼黑数据科学研究所）
R2S100K: Road-Region Segmentation Dataset For Semi-Supervised Autonomous Driving in the Wild
Information Technology University (ITU)（信息技术大学）；Computer Vision Center, Universitat Autonoma Barcelona（巴塞罗那自治大学计算机视觉中心）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）；Qatar University（卡塔尔大学）
Chengdu University of Information Technology（成都信息工程大学）；Nanjing University of Science and Technology（南京理工大学）
Robust road segmentation is a key challenge in self-driving research. Though many image-based methods have been studied and high performances in dataset evaluations have been reported, developing robust and reliable road segmentation is still a major challenge. Data fusion across different sensors to improve the performance of road segmentation is widely considered an important and irreplaceable solution. In this paper, we propose a novel structure to fuse image and LiDAR point cloud in an end-to-end semantic segmentation network, in which the fusion is performed at decoder stage instead of at…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Multi-sensor fusion-based road segmentation plays an important role in the intelligent driving system since it provides a drivable area. The existing mainstream fusion method is mainly to feature fusion in the image space domain which causes the perspective compression of the road and damages the performance of the distant road. Considering the bird's eye views(BEV) of the LiDAR remains the space structure in horizontal plane, this paper proposes a bidirectional fusion network(BiFNet) to fuse the image and BEV of the point cloud. The network consists of two modules: 1) Dense space transformati…
Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving
Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences（创新移动性研究所（IIMo），英戈尔施塔特应用技术大学）；Fraunhofer Institute for Transportation and Infrastructure Systems IVI（弗劳恩霍夫交通与基础设施系统研究所IVI）
In recent years, convolutional neural network has gained popularity in many engineering applications especially for computer vision. In order to achieve better performance, often more complex structures and advanced operations are incorporated into the neural networks, which results very long inference time. For time-critical tasks such as autonomous driving and virtual reality, real-time processing is fundamental. In order to reach real-time process speed, a light-weight, high-throughput CNN architecture namely RoadNet-RT is proposed for road segmentation in this paper. It achieves 90.33% Max…
In this paper, we introduce a deep encoder-decoder network, named SalsaNet, for efficient semantic segmentation of 3D LiDAR point clouds. SalsaNet segments the road, i.e. drivable free-space, and vehicles in the scene by employing the Bird-Eye-View (BEV) image projection of the point cloud. To overcome the lack of annotated point cloud data, in particular for the road segments, we introduce an auto-labeling process which transfers automatically generated labels from the camera to LiDAR. We also explore the role of imagelike projection of LiDAR data in semantic segmentation by comparing BEV wit…
National Chiao Tung University（国立交通大学）；Industrial Technology Research Institute（工业技术研究院）
Road scene understanding is a critical component in an autonomous driving system. Although the deep learning-based road scene segmentation can achieve very high accuracy, its complexity is also very high for developing real-time applications. It is challenging to design a neural net with high accuracy and low computational complexity. To address this issue, we investigate the advantages and disadvantages of several popular CNN architectures in terms of speed, storage and segmentation accuracy. We start from the Fully Convolutional Network (FCN) with VGG, and then we study ResNet and DenseNet. …
In this paper, we address the problem of road segmentation and free space detection in the context of autonomous driving. Traditional methods either use 3-dimensional (3D) cues such as point clouds obtained from LIDAR, RADAR or stereo cameras or 2-dimensional (2D) cues such as lane markings, road boundaries and object detection. Typical 3D point clouds do not have enough resolution to detect fine differences in heights such as between road and pavement. Image based 2D cues fail when encountering uneven road textures such as due to shadows, potholes, lane markings or road restoration. We propos…
Autonomous driving is becoming a reality, yet vehicles still need to rely on complex sensor fusion to understand the scene they act in. The ability to discern static environment and dynamic entities provides a comprehension of the road layout that poses constraints to the reasoning process about moving objects. We pursue this through a GAN-based semantic segmentation inpainting model to remove all dynamic objects from the scene and focus on understanding its static components such as streets, sidewalks and buildings. We evaluate this task on the Cityscapes dataset and on a novel synthetically …
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Road curb detection is important for autonomous driving. It can be used to determine road boundaries to constrain vehicles on roads, so that potential accidents could be avoided. Most of the current methods detect road curbs online using vehicle-mounted sensors, such as cameras or 3-D Lidars. However, these methods usually suffer from severe occlusion issues. Especially in highly-dynamic traffic environments, most of the field of view is occupied by dynamic objects. To alleviate this issue, we detect road curbs offline using high-resolution aerial images in this paper. Moreover, the detected r…
Cyprus International University（塞浦路斯国际大学）；the Hong Kong University of Science and Technology（香港科技大学）；Hangzhou ATG Intelligent Technology Co. Ltd.（杭州安智智能科技有限公司）；Guolu Gaoke Engineering Technology Institute Co. Ltd.（国路高科工程技术研究院有限公司）
In this paper, a robust lane detection algorithm is proposed, where the vertical road profile of the road is estimated using dynamic programming from the v-disparity map and, based on the estimated profile, the road area is segmented. Since the lane markings are on the road area and any feature point above the ground will be a noise source for the lane detection, a mask is created for the road area to remove some of the noise for lane detection. The estimated mask is multiplied by the lane feature map in a bird's eye view (BEV). The lane feature points are extracted by using an extended versio…
OT-Drive: Out-of-Distribution Off-Road Traversable Area Segmentation via Optimal Transport
School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Shandong Pengxiang Automobile Co., Ltd（山东鹏翔汽车有限公司）
KCIS, International Institute of Information Technology Hyderabad（KCIS，海得拉巴国际信息技术学院）
Detecting small obstacles on the road is critical for autonomous driving. In this paper, we present a method to reliably detect such obstacles through a multi-modal framework of sparse LiDAR(VLP-16) and Monocular vision. LiDAR is employed to provide additional context in the form of confidence maps to monocular segmentation networks. We show significant performance gains when the context is fed as an additional input to monocular semantic segmentation frameworks. We further present a new semantic segmentation dataset to the community, comprising of over 3000 image frames with corresponding LiD…
SAM-Enhanced Segmentation on Road Datasets: Balancing Critical Classes in Autonomous Driving
Department of Mechanical and Industrial Engineering, Tallinn University of Technology（塔林技术大学机械与工业工程系）；FinEst Centre for Smart Cities, Tallinn University of Technology（塔林技术大学智能城市研究中心）；Department of Computer Science and Engineering, Universitas Mercatorum（默卡托姆大学计算机科学与工程系）；Department of Computer Science and Engineering, Chalmers University of Technology（挑战者技术大学计算机科学与工程系）；University of Gothenburg（哥德堡大学）
Worcester Polytechnic Institute（伍斯特理工学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）
This paper presents an accurate and fast algorithm for road segmentation using convolutional neural network (CNN) and gated recurrent units (GRU). For autonomous vehicles, road segmentation is a fundamental task that can provide the drivable area for path planning. The existing deep neural network based segmentation algorithms usually take a very deep encoder-decoder structure to fuse pixels, which requires heavy computations, large memory and long processing time. Hereby, a CNN-GRU network model is proposed and trained to perform road segmentation using data captured by the front camera of a …
Remote sensing is extensively used in cartography. As transportation networks grow and change, extracting roads automatically from satellite images is crucial to keep maps up-to-date. Synthetic Aperture Radar satellites can provide high resolution topographical maps. However roads are difficult to identify in these data as they look visually similar to targets such as rivers and railways. Most road extraction methods on Synthetic Aperture Radar images still rely on a prior segmentation performed by classical computer vision algorithms. Few works study the potential of deep learning techniques,…
Technical Report for ICRA 2025 GOOSE 2D Semantic Segmentation Challenge: Boosting Off-Road Segmentation via Photometric Distortion and Exponential Moving Average
Daegu-Gyeongbuk Research Center, Electronics and Telecommunications Research Institute (ETRI)（大邱-庆尚研究中心，电子电信研究所）
IbnouZohr University（伊本佐赫尔大学）；Université Paris 13（巴黎第十三大学）；Sorbonne Paris Cité（巴黎城市大学）
Road signs detection and recognition in natural scenes is one of the most important tasksin the design of Intelligent Transport Systems (ITS). However, illumination changes remain a major problem. In this paper, an efficient ap-proach of road signs segmentation based on photometric invariants is proposed. This method is based on color in-formation using a hybrid distance, by exploiting the chro-matic distance and the red and blue ratio, on l Theta Phi color space which is invariant to highlight, shading and shadow changes. A comparative study is performed to demonstrate the robustness of this …
Smart automated traffic enforcement solutions have been gaining popularity in recent years. These solutions are ubiquitously used for seat-belt violation detection, red-light violation detection and speed violation detection purposes. Highly accurate license plate recognition is an indispensable part of these systems. However, general license plate recognition systems require high resolution images for high performance. In this study, we propose a novel license plate recognition method for general roadway surveillance cameras. Proposed segmentation free license plate recognition algorithm util…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
It is very useful and increasingly popular to extract accurate road centerlines from very-high-resolution (VHR) re- mote sensing imagery for various applications, such as road map generation and updating etc. There are three shortcomings of current methods: (a) Due to the noise and occlusions (owing to vehicles and trees), most road extraction methods bring in heterogeneous classification results; (b) Morphological thinning algorithm is widely used to extract road centerlines, while it pro- duces small spurs around the centerlines; (c) Many methods are ineffective to extract centerlines around…
We present an approach for road segmentation that only requires image-level annotations at training time. We leverage distant supervision, which allows us to train our model using images that are different from the target domain. Using large publicly available image databases as distant supervisors, we develop a simple method to automatically generate weak pixel-wise road masks. These are used to iteratively train a fully convolutional neural network, which produces our final segmentation model. We evaluate our method on the Cityscapes dataset, where we compare it with a fully supervised appro…
In 2020, there was a record heavy snowfall owing to climate change. In reality, 2,000 vehicles were stuck on the highway for three days. Because of the freezing of the road surface, 10 vehicles had a billiard accident. Road managers are required to provide indicators to alert drivers regarding snow cover at hazardous locations. This study proposes a deep learning application with live image post-processing to automatically calculate a snow hazard ratio indicator. First, the road surface hidden under snow is translated using a generative adversarial network, pix2pix. Second, snow-covered and ro…
Most state-of-the-art approaches to road extraction from aerial images rely on a CNN trained to label road pixels as foreground and remainder of the image as background. The CNN is usually trained by minimizing pixel-wise losses, which is less than ideal to produce binary masks that preserve the road network's global connectivity. To address this issue, we introduce an Adversarial Learning (AL) strategy tailored for our purposes. A naive one would treat the segmentation network as a generator and would feed its output along with ground-truth segmentations to a discriminator. It would then trai…
A Ground Segmentation Method Based on Point Cloud Map for Unstructured Roads
Research Institute for Frontier Science, Beihang University（北京航空航天大学前沿科学研究院）；School of Transportation Science and Engineering, Beihang University（北京航空航天大学交通科学与工程学院）
Towards a safe and comfortable driving, road scene segmentation is a rudimentary problem in camera-based advance driver assistance systems (ADAS). Despite of the great achievement of Convolutional Neural Networks (CNN) for semantic segmentation task, the high computational efforts of CNN based methods is still a challenging area. In recent work, we proposed a novel approach to utilise the advantages of CNNs for the task of road segmentation at reasonable computational effort. The runtime benefits from using irregular super pixels as basis for the input for the CNN rather than the image grid, w…
Finnish Geospatial Research Institute FGI, National Land Survey of Finland（芬兰国家土地调查局芬兰地理空间研究所）；Aalto University（阿尔托大学）
Robustness of different pattern recognition methods is one of the key challenges in autonomous driving, especially when driving in the high variety of road environments and weather conditions, such as gravel roads and snowfall. Although one can collect data from these adverse conditions using cars equipped with sensors, it is quite tedious to annotate the data for training. In this work, we address this limitation and propose a CNN-based method that can leverage the steering wheel angle information to improve the road area semantic segmentation. As the steering wheel angle data can be easily a…
Oxford Robotics Institute（牛津机器人研究所）；University of Oxford（牛津大学）
Road markings provide guidance to traffic participants and enforce safe driving behaviour, understanding their semantic meaning is therefore paramount in (automated) driving. However, producing the vast quantities of road marking labels required for training state-of-the-art deep networks is costly, time-consuming, and simply infeasible for every domain and condition. In addition, training data retrieved from virtual worlds often lack the richness and complexity of the real world and consequently cannot be used directly. In this paper, we provide an alternative approach in which new road marki…
Road segmentation in challenging domains, such as night, snow or rain, is a difficult task. Most current approaches boost performance using fine-tuning, domain adaptation, style transfer, or by referencing previously acquired imagery. These approaches share one or more of three significant limitations: a reliance on large amounts of annotated training data that can be costly to obtain, both anticipation of and training data from the type of environmental conditions expected at inference time, and/or imagery captured from a previous visit to the location. In this research, we remove these restr…
The Hong Kong University of Science and Technology（香港科技大学）
This paper presents a novel road damage detection algorithm based on unsupervised disparity map segmentation. Firstly, a disparity map is transformed by minimizing an energy function with respect to stereo rig roll angle and road disparity projection model. Instead of solving this energy minimization problem using non-linear optimization techniques, we directly find its numerical solution. The transformed disparity map is then segmented using Otus's thresholding method, and the damaged road areas can be extracted. The proposed algorithm requires no parameters when detecting road damage. The ex…
In automated driving systems (ADS) and advanced driver-assistance systems (ADAS), an efficient road segmentation is necessary to perceive the drivable region and build an occupancy map for path planning. The existing algorithms implement gigantic convolutional neural networks (CNNs) that are computationally expensive and time consuming. In this paper, we introduced distributed LSTM, a neural network widely used in audio and video processing, to process rows and columns in images and feature maps. We then propose a new network combining the convolutional and distributed LSTM layers to solve the…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Qatar Computing Research Institute, HBKU（卡塔尔计算研究所，哈马德·本·哈利法大学）
Mapping road networks is currently both expensive and labor-intensive. High-resolution aerial imagery provides a promising avenue to automatically infer a road network. Prior work uses convolutional neural networks (CNNs) to detect which pixels belong to a road (segmentation), and then uses complex post-processing heuristics to infer graph connectivity. We show that these segmentation methods have high error rates because noisy CNN outputs are difficult to correct. We propose RoadTracer, a new method to automatically construct accurate road network maps from aerial images. RoadTracer uses an i…
The disparity information provided by stereo cameras has enabled advanced driver assistance systems to estimate road area more accurately and effectively. In this paper, a novel disparity transformation algorithm is proposed to extract road areas from dense disparity maps by making the disparity value of the road pixels become similar. The transformation is achieved using two parameters: roll angle and fitted disparity value with respect to each row. To achieve a better processing efficiency, golden section search and dynamic programming are utilised to estimate the roll angle and the fitted d…
Semantic segmentation of road scenes is one of the key technologies for realizing autonomous driving scene perception, and the effectiveness of deep Convolutional Neural Networks(CNNs) for this task has been demonstrated. State-of-art CNNs for semantic segmentation suffer from excessive computations as well as large-scale training data requirement. Inspired by the ideas of Fine-tuning-based Transfer Learning (FTT) and feature-based knowledge distillation, we propose a new knowledge distillation method for cross-domain knowledge transference and efficient data-insufficient network training, nam…
This paper presents the preliminary findings of a semi-supervised segmentation method for extracting roads from sattelite images. Artificial Neural Networks and image segmentation methods are among the most successful methods for extracting road data from satellite images. However, these models require large amounts of training data from different regions to achieve high accuracy rates. In cases where this data needs to be of more quantity or quality, it is a standard method to train deep neural networks by transferring knowledge from annotated data obtained from different sources. This study …
The Hong Kong University of Science and Technology（香港科技大学）
The segmentation of drivable areas and road anomalies are critical capabilities to achieve autonomous navigation for robotic wheelchairs. The recent progress of semantic segmentation using deep learning techniques has presented effective results. However, the acquisition of large-scale datasets with hand-labeled ground truth is time-consuming and labor-intensive, making the deep learning-based methods often hard to implement in practice. We contribute to the solution of this problem for the task of drivable area and road anomaly segmentation by proposing a self-supervised learning approach. We…
