Collaborative multi-robot perception provides multiple views of an environment, offering varying perspectives to collaboratively understand the environment even when individual robots have poor points of view or when occlusions are caused by obstacles. These multiple observations must be intelligently fused for accurate recognition, and relevant observations need to be selected in order to allow unnecessary robots to continue on to observe other targets. This research problem has not been well studied in the literature yet. In this paper, we propose a novel approach to collaborative multi-robot perception that simultaneously integrates view selection, feature selection, and object recognition into a unified regularized optimization formulation, which uses sparsity-inducing norms to identify the robots with the most representative views and the modalities with the most discriminative features. As our optimization formulation is hard to solve due to the introduced non-smooth norms, we implement a new iterative optimization algorithm, which is guaranteed to converge to the optimal solution. We evaluate our approach through a case-study in simulation and on a physical multi-robot system. Experimental results demonstrate that our approach enables effective collaborative perception through accurate object recognition and effective view and feature selection.
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Collaborative perception has recently shown great potential to improve perception capabilities over single-agent perception. Existing collaborative perception methods usually consider an ideal communication environment. However, in practice, the communication system inevitably suffers from latency issues, causing potential performance degradation and high risks in safety-critical applications, such as autonomous driving. To mitigate the effect caused by the inevitable latency, from a machine learning perspective, we present the first latency-aware collaborative perception system, which activel…
Perception is one of the crucial module of the autonomous driving system, which has made great progress recently. However, limited ability of individual vehicles results in the bottleneck of improvement of the perception performance. To break through the limits of individual perception, collaborative perception has been proposed which enables vehicles to share information to perceive the environments beyond line-of-sight and field-of-view. In this paper, we provide a review of the related work about the promising collaborative perception technology, including introducing the fundamental concep…
CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments
New York University（纽约大学）；Tandon School of Engineering（坦登工程学院）；U.S. Army Combat Capabilities Development Command（美国陆军作战能力发展司令部）；Army Research Laboratory（陆军研究实验室）
New York University（纽约大学）；University of Southern California（南加州大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vehicle-to-everything (V2X) communication techniques enable the collaboration between vehicles and many other entities in the neighboring environment, which could fundamentally improve the perception system for autonomous driving. However, the lack of a public dataset significantly restricts the research progress of collaborative perception. To fill this gap, we present V2X-Sim, a comprehensive simulated multi-agent perception dataset for V2X-aided autonomous driving. V2X-Sim provides: (1) \hl{multi-agent} sensor recordings from the road-side unit (RSU) and multiple vehicles that enable collab…
Multi-robot systems such as swarms of aerial robots are naturally suited to offer additional flexibility, resilience, and robustness in several tasks compared to a single robot by enabling cooperation among the agents. To enhance the autonomous robot decision-making process and situational awareness, multi-robot systems have to coordinate their perception capabilities to collect, share, and fuse environment information among the agents in an efficient and meaningful way such to accurately obtain context-appropriate information or gain resilience to sensor noise or failures. In this paper, we p…
Collaborative Perception Datasets for Autonomous Driving: A Review
School of Mechanical and Electrical Engineering, China University of Mining and Technology (Beijing)（中国矿业大学（北京）机械与电子工程学院）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（哈尔滨工业大学机器人系统国家重点实验室）；State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（清华大学智能绿色车辆与移动系统国家重点实验室）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；School of Mechatronics Engineering, Harbin Institute of Technology（哈尔滨工业大学机械电子工程学院）；Department of Electronic & Electrical Engineering, University of Bath（巴斯大学电子与电气工程系）
CERF: Communication-Efficient and Retraining-Free Collaborative Perception
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
CoBEVMoE: Heterogeneity-aware Feature Fusion with Dynamic Mixture-of-Experts for Collaborative Perception
School of Computer Science and Electronic Engineering, Hunan University（计算机科学与电子工程学院，湖南大学）；School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
School of Information, Renmin University of China（中国人民大学信息学院）；Sony Research and Development Center China（索尼（中国）研发有限公司）；National University of Defense Technology（国防科技大学）
CoDS: Enhancing Collaborative Perception in Heterogeneous Scenarios via Domain Separation
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education and School of Computer Science and Technology, Beijing Jiaotong University（大数据与人工智能交通实验室（北京交通大学）及计算机科学与技术学院，北京交通大学）
Toyota Motor North America（丰田汽车北美公司）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；University of Maryland, College Park（马里兰大学帕克分校）；Colorado School of Mines（科罗拉多矿业学院）
Correspondence identification (CoID) is an essential component for collaborative perception in multi-robot systems, such as connected autonomous vehicles. The goal of CoID is to identify the correspondence of objects observed by multiple robots in their own field of view in order for robots to consistently refer to the same objects. CoID is challenging due to perceptual aliasing, object non-covisibility, and noisy sensing. In this paper, we introduce a novel deep masked graph matching approach to enable CoID and address the challenges. Our approach formulates CoID as a graph matching problem a…
CU-Multi: A Dataset for Multi-Robot Collaborative Perception
Autonomous Robotics and Perception Group at the University of Colorado Boulder（科罗拉多大学波尔得分校自主机器人与感知组）；Robot Perception Lab at Carnegie Mellon University（卡内基梅隆大学机器人感知实验室）；Aerospace Controls Laboratory at Massachusetts Institute of Technology（麻省理工学院航空航天控制实验室）；Computer Science and Artificial Intelligence Laboratory at Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；Human-Aware Robotic Exploration Lab at University of California Santa Cruz（加州大学圣克ruz分校人感知机器人探索实验室）
University of Turku（图尔库大学）；Tampere University（坦佩雷大学）
Autonomous or teleoperated robots have been playing increasingly important roles in civil applications in recent years. Across the different civil domains where robots can support human operators, one of the areas where they can have more impact is in search and rescue (SAR) operations. In particular, multi-robot systems have the potential to significantly improve the efficiency of SAR personnel with faster search of victims, initial assessment and mapping of the environment, real-time monitoring and surveillance of SAR operations, or establishing emergency communication networks, among other …
Is Intermediate Fusion All You Need for UAV-based Collaborative Perception?
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所认知与决策智能复杂系统重点实验室）
In this paper, we propose the problem of collaborative perception, where robots can combine their local observations with those of neighboring agents in a learnable way to improve accuracy on a perception task. Unlike existing work in robotics and multi-agent reinforcement learning, we formulate the problem as one where learned information must be shared across a set of agents in a bandwidth-sensitive manner to optimize for scene understanding tasks such as semantic segmentation. Inspired by networking communication protocols, we propose a multi-stage handshake communication mechanism where th…
We present an approach for estimating a mobile robot's pose w.r.t. the allocentric coordinates of a network of static cameras using multi-view RGB images. The images are processed online, locally on smart edge sensors by deep neural networks to detect the robot and estimate 2D keypoints defined at distinctive positions of the 3D robot model. Robot keypoint detections are synchronized and fused on a central backend, where the robot's pose is estimated via multi-view minimization of reprojection errors. Through the pose estimation from external cameras, the robot's localization can be initialize…
Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（工业机器人与信息研究所（西班牙国家研究委员会-加泰罗尼亚理工大学））
In this work we argue that in Human-Robot Collaboration (HRC) tasks, the Perception-Action cycle in HRC tasks can not fully explain the collaborative behaviour of the human and robot and it has to be extended to Perception-Intention-Action cycle, where Intention is a key topic. In some cases, agent Intention can be perceived or inferred by the other agent, but in others, it has to be explicitly informed to the other agent to succeed the goal of the HRC task. The Perception-Intention-Action cycle includes three basic functional procedures: Perception-Intention, Situation Awareness and Action. T…
Robust Collaborative Perception without External Localization and Clock Devices
Cooperative Medianet Innovation Center（合作中位数创新中心）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；New York University（纽约大学）；Shanghai AI Laboratory（上海人工智能实验室）
EgoRefine: Ego-Referenced Predictive Alignment and Trajectory-Conditioned Reliability-Aware Fusion for Asynchronous Collaborative Perception
Hunan University（湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；Nanjing University of Posts and Telecommunications（南京邮电大学）
