Pragmatic Heterogeneous Collaborative Perception via Generative Communication Mechanism
Southwest Jiaotong University（西南交通大学）；Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, China（可持续城市智能交通工程研究中心，教育部，中国）；Wuhan University of Technology（武汉理工大学）；City University of Hong Kong（香港城市大学）
Multi-agent collaboration enhances the perception capabilities of individual agents through information sharing. However, in real-world applications, differences in sensors and models across heterogeneous agents inevitably lead to domain gaps during collaboration. Existing approaches based on adaptation and reconstruction fail to support pragmatic heterogeneous collaboration due to two key limitations: (1) Intrusive retraining of the encoder or core modules disrupts the established semantic consistency among agents; and (2) accommodating new agents incurs high computational costs, limiting scalability. To address these challenges, we present a novel Generative Communication mechanism (GenComm) that facilitates seamless perception across heterogeneous multi-agent systems through feature generation, without altering the original network, and employs lightweight numerical alignment of spatial information to efficiently integrate new agents at minimal cost. Specifically, a tailored Deformable Message Extractor is designed to extract spatial message for each collaborator, which is then transmitted in place of intermediate features. The Spatial-Aware Feature Generator, utilizing a conditional diffusion model, generates features aligned with the ego agent's semantic space while preserving the spatial information of the collaborators. These generated features are further refined by a Channel Enhancer before fusion. Experiments conducted on the OPV2V-H, DAIR-V2X and V2X-Real datasets demonstrate that GenComm outperforms existing state-of-the-art methods, achieving an 81% reduction in both computational cost and parameter count when incorporating new agents. Our code is available at https://github.com/jeffreychou777/GenComm.
New York University（纽约大学）；University of Southern California（南加州大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vehicle-to-everything (V2X) communication techniques enable the collaboration between vehicles and many other entities in the neighboring environment, which could fundamentally improve the perception system for autonomous driving. However, the lack of a public dataset significantly restricts the research progress of collaborative perception. To fill this gap, we present V2X-Sim, a comprehensive simulated multi-agent perception dataset for V2X-aided autonomous driving. V2X-Sim provides: (1) \hl{multi-agent} sensor recordings from the road-side unit (RSU) and multiple vehicles that enable collab…
Tongji University（同济大学）；Westwell lab（西井实验室）；Technische Universität München（慕尼黑工业大学）
Multi-agent collaborative perception (MCP) has recently attracted much attention. It includes three key processes: communication for sharing, collaboration for integration, and reconstruction for different downstream tasks. Existing methods pursue designing the collaboration process alone, ignoring their intrinsic interactions and resulting in suboptimal performance. In contrast, we aim to propose a Unified Collaborative perception framework named UMC, optimizing the communication, collaboration, and reconstruction processes with the Multi-resolution technique. The communication introduces a n…
Collaborative Perception Datasets for Autonomous Driving: A Review
School of Mechanical and Electrical Engineering, China University of Mining and Technology (Beijing)（中国矿业大学（北京）机械与电子工程学院）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与移动系统国家重点实验室）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；School of Mechatronics Engineering, Harbin Institute of Technology（哈尔滨工业大学机械电子工程学院）；Department of Electronic & Electrical Engineering, University of Bath（巴斯大学电子与电气工程系）
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作媒体网络创新中心）；University of Southern California（南加州大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication. It inevitably results in a fundamental trade-off between perception performance and communication bandwidth. To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information. It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to communicate. Based on this novel spatial confidence m…
CERF: Communication-Efficient and Retraining-Free Collaborative Perception
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
CoST: Efficient Collaborative Perception From Unified Spatiotemporal Perspective
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；University of California, Los Angeles（美国加州大学洛杉矶分校）
CoDynTrust: Robust Asynchronous Collaborative Perception via Dynamic Feature Trust Modulus
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Future Science and Engineering, Soochow University（苏州大学未来科学与工程学院）；City University of Hong Kong（香港城市大学）
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Collaborative perception has recently shown great potential to improve perception capabilities over single-agent perception. Existing collaborative perception methods usually consider an ideal communication environment. However, in practice, the communication system inevitably suffers from latency issues, causing potential performance degradation and high risks in safety-critical applications, such as autonomous driving. To mitigate the effect caused by the inevitable latency, from a machine learning perspective, we present the first latency-aware collaborative perception system, which activel…
Perception is one of the crucial module of the autonomous driving system, which has made great progress recently. However, limited ability of individual vehicles results in the bottleneck of improvement of the perception performance. To break through the limits of individual perception, collaborative perception has been proposed which enables vehicles to share information to perceive the environments beyond line-of-sight and field-of-view. In this paper, we provide a review of the related work about the promising collaborative perception technology, including introducing the fundamental concep…
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机械电子工程与自动化学院）
V2X-PC: Vehicle-to-everything Collaborative Perception via Point Cluster
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University（上海交通大学合作 medianet 创新中心）；Shanghai AI Laboratory（上海人工智能实验室）；Sangfor Technologies Inc.（深信服科技股份有限公司）
CoLC: Communication-Efficient Collaborative Perception with LiDAR Completion
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education（交通运输大数据与人工智能重点实验室，教育部）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；Xiamen University（厦门大学）
Systematic Literature Review on Vehicular Collaborative Perception -- A Computer Vision Perspective
XITASO GmbH（XITASO公司）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄大学）；Technische Hochschule Ingolstadt (THI)（因戈尔施塔特技术大学）；CARISSMA Institute of Electric, Connected and Secure Mobility (C-ECOS)（CARISSMA电动、连接与安全移动研究所）；Federal University of Santa Catarina (UFSC)（圣卡塔琳娜联邦大学）
School of Information, Renmin University of China（中国人民大学信息学院）；Sony Research and Development Center China（索尼（中国）研发有限公司）；National University of Defense Technology（国防科技大学）
INSTINCT: Instance-Level Interaction Architecture for Query-Based Collaborative Perception
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Future Science and Engineering, Soochow University（苏州大学未来科学与工程学院）
INTACT: Ego-Guided Typed Sparse Evidence Retrieval for Heterogeneous Collaborative Perception
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（多谱信息智能处理国家级重点实验室，人工智能与自动化学院，华中科技大学）；Zhejiang Normal University（浙江师范大学）
CoDS: Enhancing Collaborative Perception in Heterogeneous Scenarios via Domain Separation
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education and School of Computer Science and Technology, Beijing Jiaotong University（大数据与人工智能交通实验室（北京交通大学）及计算机科学与技术学院，北京交通大学）
DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Innovation Platform for Academicians of Hainan Province（海南省院士创新平台）；Duke Kunshan University（昆山杜克大学）；Chongqing Changan Automobile Co., Ltd.（重庆长安汽车股份有限公司）
