道路分割是自动驾驶和智能机器人系统的基本感知任务，需要高精度和实时推理，特别是在资源受限的边缘设备上部署时。现有的多模态道路分割方法通常依赖重型基于Transformer的编码器以达到最先进的性能，但其巨大的计算成本阻碍了在嵌入式平台上的实时部署。为解决这一困境，我们提出了LiteViLNet，一种轻量级多模态网络，融合RGB纹理信息和LiDAR几何信息用于高效道路分割。具体来说，我们设计了双流轻量级编码器和深度可分离卷积，以最小的参数从两种模态中提取层次特征。我们进一步提出了多尺度特征融合模块（MSFM）以促进不同层次的跨模态交互，以及一个大核桥模块以线性复杂度捕获长距离依赖。在KITTI道路数据集和实际应用上的大量实验表明，LiteViLNet在准确性和效率之间取得了有希望的平衡。值得注意的是，仅用14.04M参数，我们的模型达到了96.36%的MaxF分数，在所有基于CNN的方法中排名最佳，并与更大的基于Transformer的模型相当，在RTX 4060 Ti上模型推理速度为163.79 FPS（在Jetson Orin NX上为22.18 FPS）。它在推理速度上优于许多重型方法，同时保持高度竞争的准确性，充分验证了LiteViLNet在自动驾驶和智能机器人中实时嵌入式部署的潜力。
英文摘要：
Road segmentation is a fundamental perception task for autonomous driving and mobile robotics, where both appearance and geometric cues must be processed under edge-computing constraints. Existing multi-modal approaches often improve accuracy with large encoders or expensive global interaction, which limits their use on embedded platforms. We present \textbf{LiteViLNet}, a lightweight RGB-geometry fusion network that combines a MobileNetV3 RGB encoder with a 0.12M-parameter depth-wise-separable geometry encoder. A multi-scale feature fusion module performs modality-specific enhancement, global-query cross-modal interaction, and adaptive gating, while a depth-wise large-kernel bridge enlarges the contextual support of the deepest representation with low overhead. The resulting U-Net-style decoder uses deep supervision only during training. On the KITTI Road benchmark, the 14.04M-parameter full model obtains $97.23\pm0.15\%$ MaxF. On the held-out ORFD test set under the released OFF-Net evaluation protocol, the full model achieves $96.74\pm0.09\%$ F-score and $93.68\pm0.18\%$ IoU. On a Jetson Orin NX, model-only PyTorch FP16 inference reaches $22.18\pm0.21$ FPS; a separate TensorRT FP16 measurement reaches $68.73\pm0.06$ FPS on the Jetson. Camera-depth adaptations and perception-and-control demonstrations on three heterogeneous robot platforms further illustrate the portability of the dual-stream design.
The University of Adelaide（阿德莱德大学）；Northwestern Polytechnical University（西北工业大学）；Intellifusion（云天励飞）
Road detection is a critically important task for self-driving cars. By employing LiDAR data, recent works have significantly improved the accuracy of road detection. Relying on LiDAR sensors limits the wide application of those methods when only cameras are available. In this paper, we propose a novel road detection approach with RGB being the only input during inference. Specifically, we exploit pseudo-LiDAR using depth estimation, and propose a feature fusion network where RGB and learned depth information are fused for improved road detection. To further optimize the network structure and …
In this paper, we introduce a deep encoder-decoder network, named SalsaNet, for efficient semantic segmentation of 3D LiDAR point clouds. SalsaNet segments the road, i.e. drivable free-space, and vehicles in the scene by employing the Bird-Eye-View (BEV) image projection of the point cloud. To overcome the lack of annotated point cloud data, in particular for the road segments, we introduce an auto-labeling process which transfers automatically generated labels from the camera to LiDAR. We also explore the role of imagelike projection of LiDAR data in semantic segmentation by comparing BEV wit…
For autonomous vehicles, an accurate calibration for LiDAR and camera is a prerequisite for multi-sensor perception systems. However, existing calibration techniques require either a complicated setting with various calibration targets, or an initial calibration provided beforehand, which greatly impedes their applicability in large-scale autonomous vehicle deployment. To tackle these issues, we propose a novel method to calibrate the extrinsic parameter for LiDAR and camera in road scenes. Our method introduces line features from static straight-line-shaped objects such as road lanes and pole…
University of California, Berkeley（加利福尼亚大学伯克利分校）；Ulm University（乌尔姆大学）
Detecting dynamic objects and predicting static road information such as drivable areas and ground heights are crucial for safe autonomous driving. Previous works studied each perception task separately, and lacked a collective quantitative analysis. In this work, we show that it is possible to perform all perception tasks via a simple and efficient multi-task network. Our proposed network, LidarMTL, takes raw LiDAR point cloud as inputs, and predicts six perception outputs for 3D object detection and road understanding. The network is based on an encoder-decoder architecture with 3D sparse co…
RS2AD-LiDAR: End-to-End Autonomous Driving LiDAR Data Generation from Roadside Sensor Observations
State Key Laboratory of Intelligent Green Vehicle and Mobility（智能绿色车辆与移动国家重点实验室）；School of Vehicle and Mobility（车辆与移动学院）；College of Artificial Intelligence（人工智能学院）；Logic & Silicon AI Studio；School of Instrumentation and Optoelectronic Engineering（仪器科学与光电工程学院）
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Sensor-based environmental perception is a crucial part of the autonomous driving system. In order to get an excellent perception of the surrounding environment, an intelligent system would configure multiple LiDARs (3D Light Detection and Ranging) to cover the distant and near space of the car. The precision of perception relies on the quality of sensor calibration. This research aims at developing an accurate, automatic, and robust calibration strategy for multiple LiDAR systems in the general road scene. We thus propose CROON (automatiC multi-LiDAR CalibratiOn and Refinement method in rOad …
RS2AD: End-to-End Autonomous Driving Data Generation from Roadside Sensor Observations
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与移动系统国家重点实验室）；School of Instrumentation and Optoelectronic Engineering, BeiHang University（北航仪器与光电工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）
Chengdu University of Information Technology（成都信息工程大学）；Nanjing University of Science and Technology（南京理工大学）
Robust road segmentation is a key challenge in self-driving research. Though many image-based methods have been studied and high performances in dataset evaluations have been reported, developing robust and reliable road segmentation is still a major challenge. Data fusion across different sensors to improve the performance of road segmentation is widely considered an important and irreplaceable solution. In this paper, we propose a novel structure to fuse image and LiDAR point cloud in an end-to-end semantic segmentation network, in which the fusion is performed at decoder stage instead of at…
Peking University（北京大学）；China North Vehicle Research Institute（中国北方车辆研究所）
We propose a method for off-road drivable area extraction using 3D LiDAR data with the goal of autonomous driving application. A specific deep learning framework is designed to deal with the ambiguous area, which is one of the main challenges in the off-road environment. To reduce the considerable demand for human-annotated data for network training, we utilize the information from vast quantities of vehicle paths and auto-generated obstacle labels. Using these autogenerated annotations, the proposed network can be trained using weakly supervised or semi-supervised methods, which can achieve b…
KCIS, International Institute of Information Technology Hyderabad（KCIS，海得拉巴国际信息技术学院）
Detecting small obstacles on the road is critical for autonomous driving. In this paper, we present a method to reliably detect such obstacles through a multi-modal framework of sparse LiDAR(VLP-16) and Monocular vision. LiDAR is employed to provide additional context in the form of confidence maps to monocular segmentation networks. We show significant performance gains when the context is fed as an additional input to monocular semantic segmentation frameworks. We further present a new semantic segmentation dataset to the community, comprising of over 3000 image frames with corresponding LiD…
Oxford Robotics Institute (ORI)（牛津机器人研究所）；University of Oxford（牛津大学）
Road boundaries, or curbs, provide autonomous vehicles with essential information when interpreting road scenes and generating behaviour plans. Although curbs convey important information, they are difficult to detect in complex urban environments (in particular in comparison to other elements of the road such as traffic signs and road markings). These difficulties arise from occlusions by other traffic participants as well as changing lighting and/or weather conditions. Moreover, road boundaries have various shapes, colours and structures while motion planning algorithms require accurate and …
Accurate pose estimation is a fundamental ability that all mobile robots must posses in order to traverse robustly in a given environment. Much like a human, this ability is dependent on the robot's understanding of a given scene. For Autonomous Vehicles (AV's), detailed 3D maps created beforehand are widely used to augment the perceptive abilities and estimate pose based on current sensor measurements. This approach however is less suited for rural communities that are sparsely connected and cover large areas. To deal with the challenge of localizing a vehicle in a rural setting, this paper p…
Tongji University（同济大学）；The Key Laboratory of Embedded System and Service Computing, Ministry of Education（教育部嵌入式系统与服务计算重点实验室）；Department of Computer Science and Technology, School of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院计算机科学与技术系）；School of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；The Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University, Shanghai（同济大学教育部嵌入式系统与服务计算重点实验室（上海））；Department of Computer Science and Technology, School of Electronics and Information Engineering, Tongji University, Shanghai（同济大学电子与信息工程学院计算机科学与技术系（上海））
In this paper, we studied a SLAM method for vector-based road structure mapping using multi-beam LiDAR. We propose to use the polyline as the primary mapping element instead of grid cell or point cloud, because the vector-based representation is precise and lightweight, and it can directly generate vector-based High-Definition (HD) driving map as demanded by autonomous driving systems. We explored: 1) the extraction and vectorization of road structures based on local probabilistic fusion. 2) the efficient vector-based matching between frames of road structures. 3) the loop closure and optimiza…
RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving
Jet Propulsion Laboratory (JPL)（喷气推进实验室）；California Institute of Technology (Caltech)（加州理工学院）；Swiss Federal Institute of Technology (ETH Zürich)（苏黎世联邦理工学院）
University of Science and Technology of China（中国科学技术大学）；Google Cloud（谷歌云）
Recently, it has become popular to deploy sensors such as LiDARs on the roadside to monitor the passing traffic and assist autonomous vehicle perception. Unlike autonomous vehicle systems, roadside sensors are usually affiliated with different subsystems and lack synchronization both in time and space. Calibration is a key technology which allows the central server to fuse the data generated by different location infrastructures, which can deliver improve the sensing range and detection robustness. Unfortunately, existing calibration algorithms often assume that the LiDARs are significantly ov…
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Detection of road curbs is an essential capability for autonomous driving. It can be used for autonomous vehicles to determine drivable areas on roads. Usually, road curbs are detected on-line using vehicle-mounted sensors, such as video cameras and 3-D Lidars. However, on-line detection using video cameras may suffer from challenging illumination conditions, and Lidar-based approaches may be difficult to detect far-away road curbs due to the sparsity issue of point clouds. In recent years, aerial images are becoming more and more worldwide available. We find that the visual appearances betwee…
Automated Digital Twin Construction for Highway Scenarios Using LiDAR Point Clouds and OpenStreetMap
Institute of Automotive Engineering, Graz University of Technology（格拉茨技术大学汽车工程研究所）；School of Intelligent Connected Vehicle, Hubei University of Automotive Technology（湖北汽车工业学院智能网联汽车学院）；Professorship of Autonomous Vehicle Systems, Technical University of Munich（慕尼黑工业大学自动驾驶系统教席）
Reliable knowledge of road boundaries is critical for autonomous vehicle navigation. We propose a robust curb detection and filtering technique based on the fusion of camera semantics and dense lidar point clouds. The lidar point clouds are collected by fusing multiple lidars for robust feature detection. The camera semantics are based on a modified EfficientNet architecture which is trained with labeled data collected from onboard fisheye cameras. The point clouds are associated with the closest curb segment with $L_2$-norm analysis after projecting into the image space with the fisheye model…
Nanjing University of Posts and Telecommunications（南京邮电大学）；Nanjing University of Science and Technology（南京理工大学）
Robust road detection is a key challenge in safe autonomous driving. Recently, with the rapid development of 3D sensors, more and more researchers are trying to fuse information across different sensors to improve the performance of road detection. Although many successful works have been achieved in this field, methods for data fusion under deep learning framework is still an open problem. In this paper, we propose a Siamese deep neural network based on FCN-8s to detect road region. Our method uses data collected from a monocular color camera and a Velodyne-64 LiDAR sensor. We project the LiD…
Southern University of Science and Technology（南方科技大学）；Jimei University（集美大学）
Lidar point cloud distortion from moving object is an important problem in autonomous driving, and recently becomes even more demanding with the emerging of newer lidars, which feature back-and-forth scanning patterns. Accurately estimating moving object velocity would not only provide a tracking capability but also correct the point cloud distortion with more accurate description of the moving object. Since lidar measures the time-of-flight distance but with a sparse angular resolution, the measurement is precise in the radial measurement but lacks angularly. Camera on the other hand provides…
In this work, we present a detailed comparison of ten different 3D LiDAR sensors, covering a range of manufacturers, models, and laser configurations, for the tasks of mapping and vehicle localization, using as common reference the Normal Distributions Transform (NDT) algorithm implemented in the self-driving open source platform Autoware. LiDAR data used in this study is a subset of our LiDAR Benchmarking and Reference (LIBRE) dataset, captured independently from each sensor, from a vehicle driven on public urban roads multiple times, at different times of the day. In this study, we analyze t…
Technical University of Munich（慕尼黑工业大学）；University of Bonn（波恩大学）
Temporal semantic scene understanding is critical for self-driving cars or robots operating in dynamic environments. In this paper, we propose 4D panoptic LiDAR segmentation to assign a semantic class and a temporally-consistent instance ID to a sequence of 3D points. To this end, we present an approach and a point-centric evaluation metric. Our approach determines a semantic class for every point while modeling object instances as probability distributions in the 4D spatio-temporal domain. We process multiple point clouds in parallel and resolve point-to-instance associations, effectively all…
Finnish Geospatial Research Institute FGI, National Land Survey of Finland（芬兰国家土地调查局芬兰地理空间研究所（FGI））；Aalto University School of Science（阿尔托大学理学院）
Autonomous driving is challenging in adverse road and weather conditions in which there might not be lane lines, the road might be covered in snow and the visibility might be poor. We extend the previous work on end-to-end learning for autonomous steering to operate in these adverse real-life conditions with multimodal data. We collected 28 hours of driving data in several road and weather conditions and trained convolutional neural networks to predict the car steering wheel angle from front-facing color camera images and lidar range and reflectance data. We compared the CNN model performances…
Robert Bosch GmbH（罗伯特·博世有限公司）；Ulm University（乌尔姆大学）
To assure that an autonomous car is driving safely on public roads, its object detection module should not only work correctly, but show its prediction confidence as well. Previous object detectors driven by deep learning do not explicitly model uncertainties in the neural network. We tackle with this problem by presenting practical methods to capture uncertainties in a 3D vehicle detector for Lidar point clouds. The proposed probabilistic detector represents reliable epistemic uncertainty and aleatoric uncertainty in classification and localization tasks. Experimental results show that the ep…
In this work, we present LIBRE: LiDAR Benchmarking and Reference, a first-of-its-kind dataset featuring 10 different LiDAR sensors, covering a range of manufacturers, models, and laser configurations. Data captured independently from each sensor includes three different environments and configurations: static targets, where objects were placed at known distances and measured from a fixed position within a controlled environment; adverse weather, where static obstacles were measured from a moving vehicle, captured in a weather chamber where LiDARs were exposed to different conditions (fog, rain…
Object detection and motion parameters estimation are crucial tasks for self-driving vehicle safe navigation in a complex urban environment. In this work we propose a novel real-time approach of temporal context aggregation for motion detection and motion parameters estimation based on 3D point cloud sequence. We introduce an ego-motion compensation layer to achieve real-time inference with performance comparable to a naive odometric transform of the original point cloud sequence. Not only is the proposed architecture capable of estimating the motion of common road participants like vehicles o…
We present a robust and precise localization system that achieves centimeter-level localization accuracy in disparate city scenes. Our system adaptively uses information from complementary sensors such as GNSS, LiDAR, and IMU to achieve high localization accuracy and resilience in challenging scenes, such as urban downtown, highways, and tunnels. Rather than relying only on LiDAR intensity or 3D geometry, we make innovative use of LiDAR intensity and altitude cues to significantly improve localization system accuracy and robustness. Our GNSS RTK module utilizes the help of the multi-sensor fus…
We present a LiDAR-based and real-time capable 3D perception system for automated driving in urban domains. The hierarchical system design is able to model stationary and movable parts of the environment simultaneously and under real-time conditions. Our approach extends the state of the art by innovative in-detail enhancements for perceiving road users and drivable corridors even in case of non-flat ground surfaces and overhanging or protruding elements. We describe a runtime-efficient pointcloud processing pipeline, consisting of adaptive ground surface estimation, 3D clustering and motion c…
Chemnitz University of Technology（开姆尼茨工业大学）；Drexel University（德雷塞尔大学）
Autonomous driving is a rapidly evolving technology. Autonomous vehicles are capable of sensing their environment and navigating without human input through sensory information such as radar, lidar, GNSS, vehicle odometry, and computer vision. This sensory input provides a rich dataset that can be used in combination with machine learning models to tackle multiple problems in supervised settings. In this paper we focus on road detection through gray-scale images as the sole sensory input. Our contributions are twofold: first, we introduce an annotated dataset of urban roads for machine learnin…
LiDARs are usually more accurate than cameras in distance measuring. Hence, there is strong interest to apply LiDARs in autonomous driving. Different existing approaches process the rich 3D point clouds for object detection, tracking and recognition. These methods generally require two initial steps: (1) filter points on the ground plane and (2) cluster non-ground points into objects. This paper proposes a field-tested fast 3D point cloud segmentation method for these two steps. Our specially designed algorithms allow instantly process raw LiDAR data packets, which significantly reduce the pro…
Curbs are one of the essential elements of urban and highway traffic environments. Robust curb detection provides road structure information for motion planning in an autonomous driving system. Commonly, video cameras and 3D LiDARs are mounted on autonomous vehicles for curb detection. However, camera-based methods suffer from challenging illumination conditions. During the long period of time before wide application of Deep Neural Network (DNN) with point clouds, LiDAR-based curb detection methods are based on hand-crafted features, which suffer from poor detection in some complex scenes. Rec…
With the development of autonomous driving, the improvement of autonomous driving technology for individual vehicles has reached the bottleneck. The advancement of vehicle-road cooperation autonomous driving technology can expand the vehicle's perception range, supplement the perception blind area and improve the perception accuracy, to promote the development of autonomous driving technology and achieve vehicle-road integration. This project mainly uses lidar to develop data fusion schemes to realize the sharing and combination of vehicle and road equipment data and achieve the detection and …
LIDAR has become an important part of many autonomous vehicles with its advantages on distance measurement and obstacle detection. LIDAR produces point clouds which have important information about surrounding environment. In this paper, we collected trajectory data on a two lane urban road using a Velodyne VLP-16 Lidar. Due to dynamic nature of data collection and limited range of the sensor, some of these trajectories have missing points or gaps. In this paper, we propose a novel method for recovery of missing vehicle trajectory data points using microscopic traffic flow models. While short …
