Railway LiDAR semantic segmentation based on intelligent semi-automated data annotation
Fraunhofer Institute for Open Communication Technologies (FOKUS)（弗劳恩霍夫开放通信技术研究所）；Daimler Center for Automotive Information Technology Innovations (DCAITI)（戴姆勒汽车信息技术创新中心）
自动驾驶车辆依赖于对环境准确且鲁棒的感知。与自动驾驶汽车类似，高度自动化的列车也需要环境感知。尽管汽车领域有许多基于相机或 LiDAR 传感器的研究，但针对自动化列车该任务的研究成果仍然很少。此外，目前尚无用于铁路环境 3D LiDAR 语义分割的公开数据集或已描述的方法。因此，我们提出一种基于 2DPass 网络架构的逐点 3D 语义分割方法，该方法联合使用扫描数据与图像。此外，我们提出一种半自动智能数据标注方法，用于高效且准确地为在德国铁路轨道上录制的所需数据集进行标注。为了在标注扫描数据数量仍然较少的情况下提升性能，我们采用主动学习方法来智能地为训练数据集选择扫描数据。我们的贡献包括三个方面：标注包含铁路环境相机和 LiDAR 数据的轨道数据，利用图像分割网络对原始 LiDAR 点云进行迁移标注，以及利用主动学习高效训练最先进的 3D LiDAR 语义分割网络。训练后的网络取得了良好的分割结果，在 9 个类别上的平均 IoU 为 71.48%。
英文摘要：
Automated vehicles rely on an accurate and robust perception of the environment. Similarly to automated cars, highly automated trains require an environmental perception. Although there is a lot of research based on either camera or LiDAR sensors in the automotive domain, very few contributions for this task exist yet for automated trains. Additionally, no public dataset or described approach for a 3D LiDAR semantic segmentation in the railway environment exists yet. Thus, we propose an approach for a point-wise 3D semantic segmentation based on the 2DPass network architecture using scans and images jointly. In addition, we present a semi-automated intelligent data annotation approach, which we use to efficiently and accurately label the required dataset recorded on a railway track in Germany. To improve performance despite a still small number of labeled scans, we apply an active learning approach to intelligently select scans for the training dataset. Our contributions are threefold: We annotate rail data including camera and LiDAR data from the railway environment, transfer label the raw LiDAR point clouds using an image segmentation network, and train a state-of-the-art 3D LiDAR semantic segmentation network efficiently leveraging active learning. The trained network achieves good segmentation results with a mean IoU of 71.48% of 9 classes.
Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）
Automated inspection and detection of foreign objects on railways is important for rail transportation safety as it helps prevent potential accidents and trains derailment. Most existing vision-based approaches focus on the detection of frontal intrusion objects with prior labels, such as categories and locations of the objects. In reality, foreign objects with unknown categories can appear anytime on railway tracks. In this paper, we develop a semi-supervised convolutional autoencoder based framework that only requires railway track images without prior knowledge on the foreign objects in the…
Scuola Superiore Sant’Anna（圣安娜高等学校）；Hitachi Rail STS（日立铁路STS公司）
The railway industry is searching for new ways to automate a number of complex train functions, such as object detection, track discrimination, and accurate train positioning, which require the artificial perception of the railway environment through different types of sensors, including cameras, LiDARs, wheel encoders, and inertial measurement units. A promising approach for processing such sensory data is the use of deep learning models, which proved to achieve excellent performance in other application domains, as robotics and self-driving cars. However, testing new algorithms and solutions…
George Mason University（乔治梅森大学）；University of Kansas（堪萨斯大学）
In this work, we present a novel scheduling framework enabling anytime perception for deep neural network (DNN) based 3D object detection pipelines. We focus on computationally expensive region proposal network (RPN) and per-category multi-head detector components, which are common in 3D object detection pipelines, and make them deadline-aware. We propose a scheduling algorithm, which intelligently selects the subset of the components to make effective time and accuracy trade-off on the fly. We minimize accuracy loss of skipping some of the neural network sub-components by projecting previousl…
Towards Railway Domain Adaptation for LiDAR-based 3D Detection: Road-to-Rail and Sim-to-Real via SynDRA-BBox
Mobility Technologies（移动技术）；Department of Excellence in Robotics & AI, SETLabs Research GmbH（机器人与人工智能卓越部门，SETLabs Research GmbH）；Scuola Superiore Sant’Anna（圣安娜高等学院）
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Outdoor scene completion is a challenging issue in 3D scene understanding, which plays an important role in intelligent robotics and autonomous driving. Due to the sparsity of LiDAR acquisition, it is far more complex for 3D scene completion and semantic segmentation. Since semantic features can provide constraints and semantic priors for completion tasks, the relationship between them is worth exploring. Therefore, we propose an end-to-end semantic segmentation-assisted scene completion network, including a 2D completion branch and a 3D semantic segmentation branch. Specifically, the network …
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Hikvision Research Institute（海康威视研究院）
LiDAR sensor is essential to the perception system in autonomous vehicles and intelligent robots. To fulfill the real-time requirements in real-world applications, it is necessary to efficiently segment the LiDAR scans. Most of previous approaches directly project 3D point cloud onto the 2D spherical range image so that they can make use of the efficient 2D convolutional operations for image segmentation. Although having achieved the encouraging results, the neighborhood information is not well-preserved in the spherical projection. Moreover, the temporal information is not taken into consider…
CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids
Tallinn University of Technology（塔林理工大学）；FinEst Centre for Smart Cities, Tallinn University of Technology（塔林理工大学芬埃斯特智慧城市中心）；Universitas Mercatorum（梅尔卡托鲁姆大学）
LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation
China Ministry of Education (MOE) Key Laboratory of Artificial Intelligence（中国教育部人工智能重点实验室）；Artificial Intelligence Institute, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Huawei Noah’s Ark Lab（华为诺亚实验室）
SIESEF-FusionNet: Spatial Inter-correlation Enhancement and Spatially-Embedded Feature Fusion Network for LiDAR Point Cloud Semantic Segmentation
Intelligent Autonomous Systems Laboratory, Shanghai University of Electric Power（上海电力大学智能自主系统实验室）；College of Literature, Science and the Arts, University of Michigan-Ann Arbor（密歇根大学安娜堡分校文学、科学与艺术学院）
Monocular camera sensors are vital to intelligent vehicle operation and automated driving assistance and are also heavily employed in traffic control infrastructure. Calibrating the monocular camera, though, is time-consuming and often requires significant manual intervention. In this work, we present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing semantic segmentation information from images and point clouds. Our approach relies on a coarse initial measurement of the camera pose and builds on lidar sensors mounted on a vehicle with high-precisi…
VanJee Technology（万集科技）；BeiJing Academy of Artificial Intelligence（北京人工智能研究院）
As the roadside perception plays an increasingly significant role in the Connected Automated Vehicle Highway(CAVH) technologies, there are immediate needs of challenging real-world roadside datasets for bench marking and training various computer vision tasks such as 2D/3D object detection and multi-sensor fusion. In this paper, we firstly introduce a challenging BAAI-VANJEE roadside dataset which consist of LiDAR data and RGB images collected by VANJEE smart base station placed on the roadside about 4.5m high. This dataset contains 2500 frames of LiDAR data, 5000 frames of RGB images, includi…
Edge computing-based 3D perception has received attention in intelligent transportation systems (ITS) because real-time monitoring of traffic candidates potentially strengthens Vehicle-to-Everything (V2X) orchestration. Thanks to the capability of precisely measuring the depth information on surroundings from LiDAR, the increasing studies focus on lidar-based 3D detection, which significantly promotes the development of 3D perception. Few methods met the real-time requirement of edge deployment because of high computation-intensive operations. Moreover, an inconsistency problem of object detec…
Fully autonomous driving systems require fast detection and recognition of sensitive objects in the environment. In this context, intelligent vehicles should share their sensor data with computing platforms and/or other vehicles, to detect objects beyond their own sensors' fields of view. However, the resulting huge volumes of data to be exchanged can be challenging to handle for standard communication technologies. In this paper, we evaluate how using a combination of different sensors affects the detection of the environment in which the vehicles move and operate. The final objective is to i…
CogRail: Benchmarking VLMs in Cognitive Intrusion Perception for Intelligent Railway Transportation Systems
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）；Signal & Communication Research Institute, China Academy of Railway Sciences（信号与通信研究所，中国铁路科学研究院）；Beijing Huairou Academy of Parallel Sensing（北京怀柔平行感知院）；School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院，北京交通大学）；State Key Lab of Intelligent Transportation Systems, School of Transportation Science and Engineering, Beihang University（智能交通系统国家重点实验室，交通科学与工程学院，北京航空航天大学）
Transformer-Driven Multimodal Fusion for Explainable Suspiciousness Estimation in Visual Surveillance
Big Data Research and Supercomputing Division, CSIR Fourth Paradigm Institute（CSIR第四范式研究所大数据研究与超级计算部门）；Department of Electrical Engineering, Bharti School of Telecommunication, Yardi School of Artificial Intelligence, IIT Delhi（电信学院电子工程系、Yardi人工智能学院、德里理工学院）
Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains
School of Mechanical Engineering, Hubei University of Technology（湖北工业大学机械工程学院）；Hubei Key Laboratory of Modern Manufacturing Quality Engineering, Hubei University of Technology（湖北工业大学现代制造质量工程重点实验室）；School of Mechanical Science and Engineering, Huazhong University of Science and Technology（华中科技大学机械科学与工程学院）
German Centre for Rail Traffic Research（德国铁路交通研究中心）；Federal Railway Authority（联邦铁路管理局）；Digitale Schiene Deutschland（德国数字铁路）；DB Netz AG（德国铁路网股份公司）；FusionSystems GmbH（富盛系统有限公司）
LiDARDustX: A LiDAR Dataset for Dusty Unstructured Road Environments
Wuxi Intelligent Control Research Institute,HNU（无锡智能控制研究院，湖南大学）；Hunan University（湖南大学）；Tsinghua University（清华大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）
RailGoerl24: Görlitz Rail Test Center CV Dataset 2024
German Centre for Rail Traffic Research at the Federal Railway Authority（德国铁路交通研究所在联邦铁路局）；EYYES GmbH（EYYES公司）；Chair of Fundamentals of Electrical Engineering of Dresden University of Technology（德累斯顿技术大学电气工程基础教授职位）；Conrad Zuse School of Embedded Composite AI (SECAI)（康拉德·祖塞嵌入式复合人工智能学校）
LiDAR semantic segmentation provides 3D semantic information about the environment, an essential cue for intelligent systems during their decision making processes. Deep neural networks are achieving state-of-the-art results on large public benchmarks on this task. Unfortunately, finding models that generalize well or adapt to additional domains, where data distribution is different, remains a major challenge. This work addresses the problem of unsupervised domain adaptation for LiDAR semantic segmentation models. Our approach combines novel ideas on top of the current state-of-the-art approac…
Most existing perception systems rely on sensory data acquired from cameras, which perform poorly in low light and adverse weather conditions. To resolve this limitation, we have witnessed advanced LiDAR sensors become popular in perception tasks in autonomous driving applications. Nevertheless, their usage in traffic monitoring systems is less ubiquitous. We identify two significant obstacles in cost-effectively and efficiently developing such a LiDAR-based traffic monitoring system: (i) public LiDAR datasets are insufficient for supporting perception tasks in infrastructure systems, and (ii)…
The growing demand for intelligent environments unleashes an extraordinary cycle of privacy-aware applications that makes individuals' life more comfortable and safe. Examples of these applications include pedestrian tracking systems in large areas. Although the ubiquity of camera-based systems, they are not a preferable solution due to the vulnerability of leaking the privacy of pedestrians. In this paper, we introduce a novel privacy-preserving system for pedestrian tracking in smart environments using multiple distributed LiDARs of non-overlapping views. The system is designed to leverage L…
Vision-Driven 2D Supervised Fine-Tuning Framework for Bird's Eye View Perception
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 Mobility学院）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与交通全国重点实验室）
Online,Target-Free LiDAR-Camera Extrinsic Calibration via Cross-Modal Mask Matching
Tongji University（同济大学）；College of Electronics & Information Engineering（电子与信息工程学院）；Shanghai Institute of Intelligent Science and Technology（上海智能科学技术研究院）；Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）；State Key Laboratory of Intelligent Autonomous Systems（智能自主系统国家重点实验室）；Frontiers Science Center for Intelligent Autonomous Systems（智能自主系统前沿科学中心）
Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；University of Science and Technology of China（中国科学技术大学）
Predicting the future can significantly improve the safety of intelligent vehicles, which is a key component in autonomous driving. 3D point clouds accurately model 3D information of surrounding environment and are crucial for intelligent vehicles to perceive the scene. Therefore, prediction of 3D point clouds has great significance for intelligent vehicles, which can be utilized for numerous further applications. However, due to point clouds are unordered and unstructured, point cloud prediction is challenging and has not been deeply explored in current literature. In this paper, we propose a…
LIDAR (light detection and ranging) is an optical remote-sensing technique that measures the distance between sensor and object, and the reflected energy from the object. Over the years, LIDAR data has been used as the primary source of Digital Elevation Models (DEMs). DEMs have been used in a variety of applications like road extraction, hydrological modeling, flood mapping, and surface analysis. A number of studies in flooding suggest the usage of high-resolution DEMs as inputs in the applications improve the overall reliability and accuracy. Despite the importance of high-resolution DEM, ma…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Multi-sensor fusion-based road segmentation plays an important role in the intelligent driving system since it provides a drivable area. The existing mainstream fusion method is mainly to feature fusion in the image space domain which causes the perspective compression of the road and damages the performance of the distant road. Considering the bird's eye views(BEV) of the LiDAR remains the space structure in horizontal plane, this paper proposes a bidirectional fusion network(BiFNet) to fuse the image and BEV of the point cloud. The network consists of two modules: 1) Dense space transformati…
