协同感知为增强自动驾驶汽车能力和改善道路安全提供了多项优势。除车载传感器外使用路侧传感器可提高可靠性并扩展传感器范围。外部传感器为自动驾驶汽车提供更高的态势感知并防止遮挡。我们提出协同多模态融合模型CoopDet3D和感知数据集TUMTraf-V2X，用于协同3D目标检测与跟踪任务。我们的数据集包含来自5个路侧传感器和4个车载传感器的2,000个带标注点云以及5,000张带标注图像。它包括30k个带有跟踪ID的3D框以及精确的GPS和IMU数据。我们标注了8个类别，并覆盖了包含挑战性驾驶操作的遮挡场景，如交通违规、险撞事件、超车和掉头。通过多项实验，我们表明，与车辆相机-LiDAR融合模型相比，我们的CoopDet3D相机-LiDAR融合模型实现了+14.36 3D mAP的提升。最后，我们在网站https://tum-traffic-dataset.github.io/tumtraf-v2x上公开提供数据集、模型、标注工具和开发套件。
英文摘要：
Cooperative perception offers several benefits for enhancing the capabilities of autonomous vehicles and improving road safety. Using roadside sensors in addition to onboard sensors increases reliability and extends the sensor range. External sensors offer higher situational awareness for automated vehicles and prevent occlusions. We propose CoopDet3D, a cooperative multi-modal fusion model, and TUMTraf-V2X, a perception dataset, for the cooperative 3D object detection and tracking task. Our dataset contains 2,000 labeled point clouds and 5,000 labeled images from five roadside and four onboard sensors. It includes 30k 3D boxes with track IDs and precise GPS and IMU data. We labeled eight categories and covered occlusion scenarios with challenging driving maneuvers, like traffic violations, near-miss events, overtaking, and U-turns. Through multiple experiments, we show that our CoopDet3D camera-LiDAR fusion model achieves an increase of +14.36 3D mAP compared to a vehicle camera-LiDAR fusion model. Finally, we make our dataset, model, labeling tool, and dev-kit publicly available on our website: https://tum-traffic-dataset.github.io/tumtraf-v2x.
University of California, Riverside（加州大学河滨分校）；Toyota North America InfoTech Labs（丰田北美信息技术实验室）
Utilizing the latest advances in Artificial Intelligence (AI), the computer vision community is now witnessing an unprecedented evolution in all kinds of perception tasks, particularly in object detection. Based on multiple spatially separated perception nodes, Cooperative Perception (CP) has emerged to significantly advance the perception of automated driving. However, current cooperative object detection methods mainly focus on ego-vehicle efficiency without considering the practical issues of system-wide costs. In this paper, we introduce VINet, a unified deep learning-based CP network for …
University of California, Los Angeles（加州大学洛杉矶分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）；University of California, Merced（加州大学默塞德分校）
In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel vision Transformer. Specifically, we build a holistic attention model, namely V2X-ViT, to effectively fuse information across on-road agents (i.e., vehicles and infrastructure). V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention, which captures inter-agent interaction and per-agent spat…
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）
Inter-vehicle communication for autonomous vehicles (AVs) stands to provide significant benefits in terms of perception robustness. We propose a novel approach for AVs to communicate perceptual observations, tempered by trust modelling of peers providing reports. Based on the accuracy of reported object detections as verified locally, communicated messages can be fused to augment perception performance beyond line of sight and at great distance from the ego vehicle. Also presented is a new synthetic dataset which can be used to test cooperative perception. The TruPercept dataset includes unrel…
Cleveland State University（克利夫兰州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Deep learning has been widely used in the perception (e.g., 3D object detection) of intelligent vehicle driving. Due to the beneficial Vehicle-to-Vehicle (V2V) communication, the deep learning based features from other agents can be shared to the ego vehicle so as to improve the perception of the ego vehicle. It is named as Cooperative Perception in the V2V research, whose algorithms have been dramatically advanced recently. However, all the existing cooperative perception algorithms assume the ideal V2V communication without considering the possible lossy shared features because of the Lossy …
University of California at Riverside（加州大学河滨分校）；Toyota North America R&D Labs（丰田北美研发实验室）
3D object detection plays a fundamental role in enabling autonomous driving, which is regarded as the significant key to unlocking the bottleneck of contemporary transportation systems from the perspectives of safety, mobility, and sustainability. Most of the state-of-the-art (SOTA) object detection methods from point clouds are developed based on a single onboard LiDAR, whose performance will be inevitably limited by the range and occlusion, especially in dense traffic scenarios. In this paper, we propose \textit{PillarGrid}, a novel cooperative perception method fusing information from multi…
Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
organization= College of Computer；Data Science , addressline= Fuzhou University , city= Fuzhou , country= China；organization= School of Mechanical Engineering；Automation , addressline= Fuzhou University , city= Fuzhou , country= China；organization= College of Physics；Information Engineering , addressline= Fuzhou University , city= Fuzhou , country= China
University of Central Florida（中佛罗里达大学）；InfoTech Labs, Toyota Motor North America（丰田汽车北美公司InfoTech实验室）
The recent advancement in computational and communication systems has led to the introduction of high-performing neural networks and high-speed wireless vehicular communication networks. As a result, new technologies such as cooperative perception and cognition have emerged, addressing the inherent limitations of sensory devices by providing solutions for the detection of partially occluded targets and expanding the sensing range. However, designing a reliable cooperative cognition or perception system requires addressing the challenges caused by limited network resources and discrepancies bet…
University of California at Riverside（加州大学河滨分校）；Toyota Motor North America（丰田汽车北美公司）
Object detection plays a fundamental role in enabling Cooperative Driving Automation (CDA), which is regarded as the revolutionary solution to addressing safety, mobility, and sustainability issues of contemporary transportation systems. Although current computer vision technologies could provide satisfactory object detection results in occlusion-free scenarios, the perception performance of onboard sensors could be inevitably limited by the range and occlusion. Owing to flexible position and pose for sensor installation, infrastructure-based detection and tracking systems can enhance the perc…
University of Warwick（华威大学）；Jaguar Land Rover Ltd.（捷豹路虎有限公司）；University of Surrey（萨里大学）
3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the…
EgoRefine: Ego-Referenced Predictive Alignment and Trajectory-Conditioned Reliability-Aware Fusion for Asynchronous Collaborative Perception
Hunan University（湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Human-vehicle cooperative driving has become the critical technology of autonomous driving, which reduces the workload of human drivers. However, the complex and uncertain road environments bring great challenges to the visual perception of cooperative systems. And the perception characteristics of autonomous driving differ from manual driving a lot. To enhance the visual perception capability of human-vehicle cooperative driving, this paper proposed a cooperative visual perception model. 506 images of complex road and traffic scenarios were collected as the data source. Then this paper improv…
Cooperative perception allows a Connected Autonomous Vehicle (CAV) to interact with the other CAVs in the vicinity to enhance perception of surrounding objects to increase safety and reliability. It can compensate for the limitations of the conventional vehicular perception such as blind spots, low resolution, and weather effects. An effective feature fusion model for the intermediate fusion methods of cooperative perception can improve feature selection and information aggregation to further enhance the perception accuracy. We propose adaptive feature fusion models with trainable feature sele…
University of California Los Angeles（加州大学洛杉矶分校）；Cleveland State University（克利夫兰州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Northwestern University（西北大学）；Technical University of Munich（慕尼黑工业大学）；Fraunhofer Institute（弗劳恩霍夫研究所）
Modern perception systems of autonomous vehicles are known to be sensitive to occlusions and lack the capability of long perceiving range. It has been one of the key bottlenecks that prevents Level 5 autonomy. Recent research has demonstrated that the Vehicle-to-Vehicle (V2V) cooperative perception system has great potential to revolutionize the autonomous driving industry. However, the lack of a real-world dataset hinders the progress of this field. To facilitate the development of cooperative perception, we present V2V4Real, the first large-scale real-world multi-modal dataset for V2V percep…
Autonomous vehicles may make wrong decisions due to inaccurate detection and recognition. Therefore, an intelligent vehicle can combine its own data with that of other vehicles to enhance perceptive ability, and thus improve detection accuracy and driving safety. However, multi-vehicle cooperative perception requires the integration of real world scenes and the traffic of raw sensor data exchange far exceeds the bandwidth of existing vehicular networks. To the best our knowledge, we are the first to conduct a study on raw-data level cooperative perception for enhancing the detection ability of…
Vehicle-to-Everything Cooperative Perception for Autonomous Driving
College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；Momoni AI；Department of Electrical and Computer Engineering, University of Alabama in Huntsville（电气与计算机工程系，阿拉巴马大学亨茨维尔分校）；Department of Automation and Intelligent Sensing, Shanghai Jiaotong University（自动化与智能感知系，上海交通大学）；School of Engineering, Swinburne University of Technology（工程学院，斯威本技术大学）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（信息与通信研究机构，科技研究局（A*STAR））
V2U4Real: A Real-world Large-scale Dataset for Vehicle-to-UAV Cooperative Perception
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, China（福建省城市智能感知与计算重点实验室，厦门大学，中国）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China（多媒体可信感知与高效计算教育部重点实验室，厦门大学，中国）
V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perception
School of Vehicle and Mobility, Tsinghua University（车辆与移动性学院，清华大学）；Nanyang Technological University（南洋理工大学）；CUMTB（中国交通车辆技术研究所）；University of California, Los Angeles（加州大学洛杉矶分校）；Beijing Jiaotong University（北京交通大学）；ByteDance（字节跳动）
CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception
Institute for AI Industry Research, Tsinghua University（人工智能产业研究所，清华大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；School of Vehicle and Mobility, Tsinghua University（车辆与移动技术学院，清华大学）；Baidu Inc.（百度公司）
CoMamba: Real-time Cooperative Perception Unlocked with State Space Models
Texas A&M University（得克萨斯农工大学）；Cleveland State University（克利夫兰州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of California, Riverside（加利福尼亚大学河滨分校）
University of California at Riverside（加州大学河滨分校）；Toyota Motor North America（丰田汽车北美公司）；Leidos Inc.（莱多斯公司）
Perceiving the environment is one of the most fundamental keys to enabling Cooperative Driving Automation (CDA), which is regarded as the revolutionary solution to addressing the safety, mobility, and sustainability issues of contemporary transportation systems. Although an unprecedented evolution is now happening in the area of computer vision for object perception, state-of-the-art perception methods are still struggling with sophisticated real-world traffic environments due to the inevitably physical occlusion and limited receptive field of single-vehicle systems. Based on multiple spatiall…
Coop-WD: Cooperative Perception with Weighting and Denoising for Robust V2V Communication
Department of Engineering, Durham University（工程系，达勒姆大学）；Faculty of Engineering and Information Technology, University of Technology, Sydney（工程与信息技术学院，悉尼技术大学）；Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（可信自主系统研究院，南方科技大学）
Autonomous vehicles are heavily reliant upon their sensors to perfect the perception of surrounding environments, however, with the current state of technology, the data which a vehicle uses is confined to that from its own sensors. Data sharing between vehicles and/or edge servers is limited by the available network bandwidth and the stringent real-time constraints of autonomous driving applications. To address these issues, we propose a point cloud feature based cooperative perception framework (F-Cooper) for connected autonomous vehicles to achieve a better object detection precision. Not o…
CATNet: Collaborative Alignment and Transformation Network for Cooperative Perception
School of Computer Science and Technology, Tianjin University（计算机科学与技术学院，天津大学）；School of Cybersecurity, Tianjin University（网络安全学院，天津大学）；School of Future Technology, Tianjin University（未来技术学院，天津大学）
Unlocking Past Information: Temporal Embeddings in Cooperative Bird's Eye View Prediction
Department of Computer Science and AImotion Bavaria, Technische Hochschule Ingolstadt（计算机科学系和AImotion巴伐利亚，因戈尔施塔特技术大学）；Department of Electrical Engineering and AImotion Bavaria, Technische Hochschule Ingolstadt（电气工程系和AImotion巴伐利亚，因戈尔施塔特技术大学）；School of Engineering and Design, Technical University of Munich（工程与设计学院，慕尼黑技术大学）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）
