Cooperative perception can effectively enhance individual perception performance by providing additional viewpoint and expanding the sensing field. Existing cooperation paradigms are either interpretable (result cooperation) or flexible (feature cooperation). In this paper, we propose the concept of query cooperation to enable interpretable instance-level flexible feature interaction. To specifically explain the concept, we propose a cooperative perception framework, termed QUEST, which let query stream flow among agents. The cross-agent queries are interacted via fusion for co-aware instances and complementation for individual unaware instances. Taking camera-based vehicle-infrastructure perception as a typical practical application scene, the experimental results on the real-world dataset, DAIR-V2X-Seq, demonstrate the effectiveness of QUEST and further reveal the advantage of the query cooperation paradigm on transmission flexibility and robustness to packet dropout. We hope our work can further facilitate the cross-agent representation interaction for better cooperative perception in practice.
Human-vehicle cooperative driving has become the critical technology of autonomous driving, which reduces the workload of human drivers. However, the complex and uncertain road environments bring great challenges to the visual perception of cooperative systems. And the perception characteristics of autonomous driving differ from manual driving a lot. To enhance the visual perception capability of human-vehicle cooperative driving, this paper proposed a cooperative visual perception model. 506 images of complex road and traffic scenarios were collected as the data source. Then this paper improv…
Cooperative perception allows a Connected Autonomous Vehicle (CAV) to interact with the other CAVs in the vicinity to enhance perception of surrounding objects to increase safety and reliability. It can compensate for the limitations of the conventional vehicular perception such as blind spots, low resolution, and weather effects. An effective feature fusion model for the intermediate fusion methods of cooperative perception can improve feature selection and information aggregation to further enhance the perception accuracy. We propose adaptive feature fusion models with trainable feature sele…
University of California Los Angeles（加州大学洛杉矶分校）；Cleveland State University（克利夫兰州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Northwestern University（西北大学）；Technical University of Munich（慕尼黑工业大学）；Fraunhofer Institute（弗劳恩霍夫研究所）
Modern perception systems of autonomous vehicles are known to be sensitive to occlusions and lack the capability of long perceiving range. It has been one of the key bottlenecks that prevents Level 5 autonomy. Recent research has demonstrated that the Vehicle-to-Vehicle (V2V) cooperative perception system has great potential to revolutionize the autonomous driving industry. However, the lack of a real-world dataset hinders the progress of this field. To facilitate the development of cooperative perception, we present V2V4Real, the first large-scale real-world multi-modal dataset for V2V percep…
University of California, Riverside（加州大学河滨分校）；Toyota North America InfoTech Labs（丰田北美信息技术实验室）
Utilizing the latest advances in Artificial Intelligence (AI), the computer vision community is now witnessing an unprecedented evolution in all kinds of perception tasks, particularly in object detection. Based on multiple spatially separated perception nodes, Cooperative Perception (CP) has emerged to significantly advance the perception of automated driving. However, current cooperative object detection methods mainly focus on ego-vehicle efficiency without considering the practical issues of system-wide costs. In this paper, we introduce VINet, a unified deep learning-based CP network for …
Autonomous vehicles may make wrong decisions due to inaccurate detection and recognition. Therefore, an intelligent vehicle can combine its own data with that of other vehicles to enhance perceptive ability, and thus improve detection accuracy and driving safety. However, multi-vehicle cooperative perception requires the integration of real world scenes and the traffic of raw sensor data exchange far exceeds the bandwidth of existing vehicular networks. To the best our knowledge, we are the first to conduct a study on raw-data level cooperative perception for enhancing the detection ability of…
Vehicle-to-Everything Cooperative Perception for Autonomous Driving
College of Science and Engineering, James Cook University（科学与工程学院，詹姆斯库克大学）；Momoni AI；Department of Electrical and Computer Engineering, University of Alabama in Huntsville（电气与计算机工程系，阿拉巴马大学亨茨维尔分校）；Department of Automation and Intelligent Sensing, Shanghai Jiaotong University（自动化与智能感知系，上海交通大学）；School of Engineering, Swinburne University of Technology（工程学院，斯威本技术大学）；Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（信息与通信研究机构，科技研究局（A*STAR））
V2U4Real: A Real-world Large-scale Dataset for Vehicle-to-UAV Cooperative Perception
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, China（福建省城市智能感知与计算重点实验室，厦门大学，中国）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China（多媒体可信感知与高效计算教育部重点实验室，厦门大学，中国）
V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perception
School of Vehicle and Mobility, Tsinghua University（车辆与移动性学院，清华大学）；Nanyang Technological University（南洋理工大学）；CUMTB（中国交通车辆技术研究所）；University of California, Los Angeles（加州大学洛杉矶分校）；Beijing Jiaotong University（北京交通大学）；ByteDance（字节跳动）
CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception
Institute for AI Industry Research, Tsinghua University（人工智能产业研究所，清华大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；School of Vehicle and Mobility, Tsinghua University（车辆与移动技术学院，清华大学）；Baidu Inc.（百度公司）
CoMamba: Real-time Cooperative Perception Unlocked with State Space Models
Texas A&M University（得克萨斯农工大学）；Cleveland State University（克利夫兰州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of California, Riverside（加利福尼亚大学河滨分校）
University of California at Riverside（加州大学河滨分校）；Toyota Motor North America（丰田汽车北美公司）；Leidos Inc.（莱多斯公司）
Perceiving the environment is one of the most fundamental keys to enabling Cooperative Driving Automation (CDA), which is regarded as the revolutionary solution to addressing the safety, mobility, and sustainability issues of contemporary transportation systems. Although an unprecedented evolution is now happening in the area of computer vision for object perception, state-of-the-art perception methods are still struggling with sophisticated real-world traffic environments due to the inevitably physical occlusion and limited receptive field of single-vehicle systems. Based on multiple spatiall…
University of California, Los Angeles（加州大学洛杉矶分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Google Research（谷歌研究院）；University of California, Merced（加州大学默塞德分校）
In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel vision Transformer. Specifically, we build a holistic attention model, namely V2X-ViT, to effectively fuse information across on-road agents (i.e., vehicles and infrastructure). V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention, which captures inter-agent interaction and per-agent spat…
Coop-WD: Cooperative Perception with Weighting and Denoising for Robust V2V Communication
Department of Engineering, Durham University（工程系，达勒姆大学）；Faculty of Engineering and Information Technology, University of Technology, Sydney（工程与信息技术学院，悉尼技术大学）；Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（可信自主系统研究院，南方科技大学）
Autonomous vehicles are heavily reliant upon their sensors to perfect the perception of surrounding environments, however, with the current state of technology, the data which a vehicle uses is confined to that from its own sensors. Data sharing between vehicles and/or edge servers is limited by the available network bandwidth and the stringent real-time constraints of autonomous driving applications. To address these issues, we propose a point cloud feature based cooperative perception framework (F-Cooper) for connected autonomous vehicles to achieve a better object detection precision. Not o…
Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Driving
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作 Medianet 创新中心）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Warwick（华威大学）
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）
Inter-vehicle communication for autonomous vehicles (AVs) stands to provide significant benefits in terms of perception robustness. We propose a novel approach for AVs to communicate perceptual observations, tempered by trust modelling of peers providing reports. Based on the accuracy of reported object detections as verified locally, communicated messages can be fused to augment perception performance beyond line of sight and at great distance from the ego vehicle. Also presented is a new synthetic dataset which can be used to test cooperative perception. The TruPercept dataset includes unrel…
CATNet: Collaborative Alignment and Transformation Network for Cooperative Perception
School of Computer Science and Technology, Tianjin University（计算机科学与技术学院，天津大学）；School of Cybersecurity, Tianjin University（网络安全学院，天津大学）；School of Future Technology, Tianjin University（未来技术学院，天津大学）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）；Stanford University（斯坦福大学）
Optical sensors and learning algorithms for autonomous vehicles have dramatically advanced in the past few years. Nonetheless, the reliability of today's autonomous vehicles is hindered by the limited line-of-sight sensing capability and the brittleness of data-driven methods in handling extreme situations. With recent developments of telecommunication technologies, cooperative perception with vehicle-to-vehicle communications has become a promising paradigm to enhance autonomous driving in dangerous or emergency situations. We introduce COOPERNAUT, an end-to-end learning model that uses cross…
Cleveland State University（克利夫兰州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Deep learning has been widely used in the perception (e.g., 3D object detection) of intelligent vehicle driving. Due to the beneficial Vehicle-to-Vehicle (V2V) communication, the deep learning based features from other agents can be shared to the ego vehicle so as to improve the perception of the ego vehicle. It is named as Cooperative Perception in the V2V research, whose algorithms have been dramatically advanced recently. However, all the existing cooperative perception algorithms assume the ideal V2V communication without considering the possible lossy shared features because of the Lossy …
RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；China Automotive Innovation Corporation（中国汽车创新公司）；The University of Hong Kong（香港大学）
Cooperative Perception with Learning-Based V2V communications
University of Warwick（华威大学）；University of Durham（杜伦大学）；University of Technology Sydney（悉尼科技大学）；Oracle UK（英国甲骨文公司）；Southern University of Science and Technology（南方科技大学）；University of Reading（雷丁大学）
Unlocking Past Information: Temporal Embeddings in Cooperative Bird's Eye View Prediction
Department of Computer Science and AImotion Bavaria, Technische Hochschule Ingolstadt（计算机科学系和AImotion巴伐利亚，因戈尔施塔特技术大学）；Department of Electrical Engineering and AImotion Bavaria, Technische Hochschule Ingolstadt（电气工程系和AImotion巴伐利亚，因戈尔施塔特技术大学）；School of Engineering and Design, Technical University of Munich（工程与设计学院，慕尼黑技术大学）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）
University of Central Florida（中佛罗里达大学）；InfoTech Labs, Toyota Motor North America（丰田汽车北美公司InfoTech实验室）
The recent advancement in computational and communication systems has led to the introduction of high-performing neural networks and high-speed wireless vehicular communication networks. As a result, new technologies such as cooperative perception and cognition have emerged, addressing the inherent limitations of sensory devices by providing solutions for the detection of partially occluded targets and expanding the sensing range. However, designing a reliable cooperative cognition or perception system requires addressing the challenges caused by limited network resources and discrepancies bet…
MMUSV-Sim: A Perception-Oriented Simulation and Data-Generation Platform for Multi-USV Cooperative Perception
School of Intelligent Systems Engineering, Sun Yat-sen University（中山大学智能工程学院）；School of Intelligent Imagery Engineering, Beijing Film Academy（北京电影学院智能影像工程学院）
University of California at Riverside（加州大学河滨分校）；Toyota North America R&D Labs（丰田北美研发实验室）
3D object detection plays a fundamental role in enabling autonomous driving, which is regarded as the significant key to unlocking the bottleneck of contemporary transportation systems from the perspectives of safety, mobility, and sustainability. Most of the state-of-the-art (SOTA) object detection methods from point clouds are developed based on a single onboard LiDAR, whose performance will be inevitably limited by the range and occlusion, especially in dense traffic scenarios. In this paper, we propose \textit{PillarGrid}, a novel cooperative perception method fusing information from multi…
COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligence
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Baidu Inc.（百度公司）
