CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Huawei Noah’s Ark Lab（华为诺亚实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
Runjian Chen, Yao Mu, Runsen Xu, Wenqi Shao, Chenhan Jiang, Hang Xu, Zhenguo Li, Ping Luo
Unsupervised contrastive learning for indoor-scene point clouds has achieved great successes. However, unsupervised learning point clouds in outdoor scenes remains challenging because previous methods need to reconstruct the whole scene and capture partial views for the contrastive objective. This is infeasible in outdoor scenes with moving objects, obstacles, and sensors. In this paper, we propose CO^3, namely Cooperative Contrastive Learning and Contextual Shape Prediction, to learn 3D representation for outdoor-scene point clouds in an unsupervised manner. CO^3 has several merits compared to existing methods. (1) It utilizes LiDAR point clouds from vehicle-side and infrastructure-side to build views that differ enough but meanwhile maintain common semantic information for contrastive learning, which are more appropriate than views built by previous methods. (2) Alongside the contrastive objective, shape context prediction is proposed as pre-training goal and brings more task-relevant information for unsupervised 3D point cloud representation learning, which are beneficial when transferring the learned representation to downstream detection tasks. (3) As compared to previous methods, representation learned by CO^3 is able to be transferred to different outdoor scene dataset collected by different type of LiDAR sensors. (4) CO^3 improves current state-of-the-art methods on both Once and KITTI datasets by up to 2.58 mAP. We believe CO^3 will facilitate understanding LiDAR point clouds in outdoor scene.
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）；Stanford University（斯坦福大学）
Optical sensors and learning algorithms for autonomous vehicles have dramatically advanced in the past few years. Nonetheless, the reliability of today's autonomous vehicles is hindered by the limited line-of-sight sensing capability and the brittleness of data-driven methods in handling extreme situations. With recent developments of telecommunication technologies, cooperative perception with vehicle-to-vehicle communications has become a promising paradigm to enhance autonomous driving in dangerous or emergency situations. We introduce COOPERNAUT, an end-to-end learning model that uses cross…
CoLMIN: LLM-based Multi-Decision Path Negotiation for Cooperative Autonomous Driving
School of future Transportation, Chang’an University（长安大学未来交通学院）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of information, Renmin University of China（中国人民大学信息学院）；School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）
V2X-WAM: A Cooperative World Action Model for End-to-End Autonomous Driving
ITS Center, Research Institute of Highway Ministry of Transport（交通运输部公路科学研究院智能交通系统中心）；State Key Lab of Intelligent Transportation System, Research Institute of Highway Ministry of Transport（交通运输部公路科学研究院智能交通系统国家重点实验室）；Department of Civil and Environmental Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校土木与环境工程系）；Intelligent Transportation Systems Research Center, Wuhan University of Technology（武汉理工大学智能交通系统研究中心）；Engineering Research Center of Transportation Information and Safety, Ministry of Education（教育部交通信息与安全工程研究中心）；School of Transportation, Inner Mongolia University（内蒙古大学交通学院）；College of Transportation, Tongji University（同济大学交通运输工程学院）
Human-vehicle cooperative driving has become the critical technology of autonomous driving, which reduces the workload of human drivers. However, the complex and uncertain road environments bring great challenges to the visual perception of cooperative systems. And the perception characteristics of autonomous driving differ from manual driving a lot. To enhance the visual perception capability of human-vehicle cooperative driving, this paper proposed a cooperative visual perception model. 506 images of complex road and traffic scenarios were collected as the data source. Then this paper improv…
Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competition
Tsinghua University（清华大学）；Hong Kong University（香港大学）；Tongji University（同济大学）；Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Stanford University（斯坦福大学）；OpenDriveLab（OpenDrive实验室）；Technical University of Munich（慕尼黑技术大学）；Imperial College London（伦敦帝国理工学院）；Peking University（北京大学）
PerceptIn（感知创新公司）；South China University of Technology（华南理工大学）；Northwestern University（西北大学）
The infrastructure-vehicle cooperative autonomous driving approach depends on the cooperation between intelligent roads and intelligent vehicles. This approach is not only safer but also more economical compared to the traditional on-vehicle-only autonomous driving approach. In this paper, we introduce our real-world deployment experiences of cooperative autonomous driving, and delve into the details of new challenges and opportunities. Specifically, based on our progress towards commercial deployment, we follow a three-stage development roadmap of the cooperative autonomous driving approach:i…
Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作 Medianet 创新中心）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Warwick（华威大学）
V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Department of Civil and Environmental Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校土木与环境工程系）；Department of Civil and Architectural Engineering and Construction Management, University of Wyoming（怀俄明大学土木与建筑工程及施工管理系）；School of Transportation, Southeast University（东南大学交通学院）
Department of Civil and Environmental Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校土木与环境工程系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学 computing and Data Science 学院）；College of Transportation, Tongji University（同济大学交通运输学院）
Advances in Single-vehicle intelligence of automated driving have encountered significant challenges because of limited capabilities in perception and interaction with complex traffic environments. Cooperative Driving Automation~(CDA) has been considered a pivotal solution to next-generation automated driving and intelligent transportation. Though CDA has attracted much attention from both academia and industry, exploration of its potential is still in its infancy. In industry, companies tend to build their in-house data collection pipeline and research tools to tailor their needs and protect …
Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）；School of Computing, Tokyo Institute of Technology（东京工业大学计算机学院）
Cooperative intelligent transportation systems (ITS) are used by autonomous vehicles to communicate with surrounding autonomous vehicles and roadside units (RSU). Current C-ITS applications focus primarily on real-time information sharing, such as cooperative perception. In addition to real-time information sharing, self-driving cars need to coordinate their action plans to achieve higher safety and efficiency. For this reason, this study defines a vehicle's future action plan/path and designs a cooperative path-planning model at intersections using future path sharing based on the future path…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Delaware（特拉华大学）；University of Chinese Academy of Sciences（中国科学院大学）
Infrastructure-Vehicle Cooperative Autonomous Driving (IVCAD) is a new paradigm of autonomous driving, which relies on the cooperation between intelligent roads and autonomous vehicles. This paradigm has been shown to be safer and more efficient compared to the on-vehicle-only autonomous driving paradigm. Our real-world deployment data indicates that the effectiveness of IVCAD is constrained by reliability and performance of commercial communication networks. This paper targets this exact problem, and proposes INTERNEURON, a middleware to achieve high communication reliability between intellig…
Short-term future of automated driving can be imagined as a hybrid scenario in which both automated and human-driven vehicles co-exist in the same environment. In order to address the needs of such road configuration, many technology solutions such as vehicular communication and predictive control for automated vehicles have been introduced in the literature. Both aforementioned solutions rely on driving data of the human driver. In this work, we investigate the currently available driving datasets and introduce a real-world maneuver-based driving dataset that is collected during our urban dri…
Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planning
Key Laboratory of Road and Traffic Engineering of the Ministry of Education, Tongji University（教育部道路与交通工程重点实验室，同济大学）；UCLA Mobility Lab, University of California, Los Angeles（加州大学洛杉矶分校UCLA移动实验室）
King’s College London（伦敦国王学院）；Centre For Telecommunications Research Department of Engineering（电信研究中心工程系）
In this paper, we study the role that machine learning can play in cooperative driving. Given the increasing rate of connectivity in modern vehicles, and road infrastructure, cooperative driving is a promising first step in automated driving. The example scenario we explored in this paper, is coordinated lane merge, with data collection, test and evaluation all conducted in an automotive test track. The assumption is that vehicles are a mix of those equipped with communication units on board, i.e. connected vehicles, and those that are not connected. However, roadside cameras are connected and…
Toward Cooperative Driving in Mixed Traffic: An Adaptive Potential Game-Based Approach with Field Test Verification
State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（智能绿色车辆与移动国家重点实验室，清华大学）；State Key Lab of Intelligent Transportation System, School of Transportation Science and Engineering, Beihang University（智能交通运输系统实验室，北京航空航天大学交通科学与工程学院）
Recently, there are emerging many stereo matching methods for autonomous driving based on unsupervised learning. Most of them take advantage of reconstruction losses to remove dependency on disparity groundtruth. Occlusion handling is a challenging problem in stereo matching, especially for unsupervised methods. Previous unsupervised methods failed to take full advantage of geometry properties in occlusion handling. In this paper, we introduce an effective way to detect occlusion regions and propose a novel unsupervised training strategy to deal with occlusion that only uses the predicted left…
Learning-based perception and prediction modules in modern autonomous driving systems typically rely on expensive human annotation and are designed to perceive only a handful of predefined object categories. This closed-set paradigm is insufficient for the safety-critical autonomous driving task, where the autonomous vehicle needs to process arbitrarily many types of traffic participants and their motion behaviors in a highly dynamic world. To address this difficulty, this paper pioneers a novel and challenging direction, i.e., training perception and prediction models to understand open-set m…
OpenConvoy: Universal Platform for Real-World Testing of Cooperative Driving Systems
College of Engineering；Computer Science University of Central Florida Orlando, United States https://orcid.org/0009-0007-1826-373X；College of Electrical；Computer Engineering University of Central Florida Orlando, United States；Computer Science University of Central Florida Orlando, United States
Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning
Department of Computer Science at University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）；College of Transportation and Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University（同济大学交通学院及交通工程教育部重点实验室）；Department of Mechanical Engineering at the University of California, Berkeley（加州大学伯克利分校机械工程系）
In this paper, we introduce the notion of Cooperative Perception Error Models (coPEMs) towards achieving an effective and efficient integration of V2X solutions within a virtual test environment. We focus our analysis on the occlusion problem in the (onboard) perception of Autonomous Vehicles (AV), which can manifest as misdetection errors on the occluded objects. Cooperative perception (CP) solutions based on Vehicle-to-Everything (V2X) communications aim to avoid such issues by cooperatively leveraging additional points of view for the world around the AV. This approach usually requires many…
Vehicle-to-Everything Cooperative Perception for Autonomous Driving
College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；Momoni AI；Department of Electrical and Computer Engineering, University of Alabama in Huntsville（电气与计算机工程系，阿拉巴马大学亨茨维尔分校）；Department of Automation and Intelligent Sensing, Shanghai Jiaotong University（自动化与智能感知系，上海交通大学）；School of Engineering, Swinburne University of Technology（工程学院，斯威本技术大学）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（信息与通信研究机构，科技研究局（A*STAR））
With the development of autonomous driving, the improvement of autonomous driving technology for individual vehicles has reached the bottleneck. The advancement of vehicle-road cooperation autonomous driving technology can expand the vehicle's perception range, supplement the perception blind area and improve the perception accuracy, to promote the development of autonomous driving technology and achieve vehicle-road integration. This project mainly uses lidar to develop data fusion schemes to realize the sharing and combination of vehicle and road equipment data and achieve the detection and …
University of Warwick（华威大学）；Jaguar Land Rover Ltd.（捷豹路虎有限公司）；University of Surrey（萨里大学）
3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the…
College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）；School of Information Science and Engineering, Shenyang University of Technology（沈阳工业大学信息科学与工程学院）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与 mobility学院）；School of Software, Beihang University（北京航空航天大学软件学院）；School of Information and Engineering, China University of Geosciences Beijing（中国地质大学（北京）信息与工程学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models
organization= Department of Civil；Environmental Engineering, University of Wisconsin–Madison , city= Madison , state= WI , postcode= 53706 , country= USA；organization= College of Computing；Data Science, Nanyang Technological University , city= Singapore , postcode= 639798 , country= Singapore；organization= College of Transportation, Tongji University , city= Shanghai , postcode= 201804 , country= China；organization= Zachry Department of Civil；Environmental Engineering, Texas A\&M University , city= College Station , state= TX , postcode= 77840 , country= USA；organization= School of Civil；Environmental Engineering, Cornell University , city= Ithaca , state= NY , postcode= 14853 , country= USA
