Multi-agent collaborative perception (CP) is a promising paradigm for improving autonomous driving safety, particularly for vulnerable road users like pedestrians, via robust 3D perception. However, existing CP approaches often optimize for vehicle detection performance metrics, underperforming on smaller, safety-critical objects such as pedestrians, where detection failures can be catastrophic. Furthermore, previous CP methods rely on full feature exchange rather than communicating only salient features that help reduce false negatives. To this end, we present FocalComm, a novel collaborative perception framework that focuses on exchanging hard-instance-oriented features among connected collaborative agents. FocalComm consists of two key novel designs: (1) a learnable progressive hard instance mining (HIM) module to extract hard instance-oriented features per agent, and (2) a query-based feature-level (intermediate) fusion technique that dynamically weights these identified features during collaboration. We show that FocalComm outperforms state-of-the-art collaborative perception methods on two challenging real-world datasets (V2X-Real and DAIR-V2X) across both vehicle-centric and infrastructure-centric collaborative setups. FocalComm also shows a strong performance gain in pedestrian detection in V2X-Real.
School of Information, Renmin University of China（中国人民大学信息学院）；Sony Research and Development Center China（索尼（中国）研发有限公司）；National University of Defense Technology（国防科技大学）
InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University（北京大学深圳研究生院广东省超高清沉浸式媒体技术重点实验室）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）
University of Florida（佛罗里达大学）；Purdue University（普渡大学）；The University of Texas at El Paso（德克萨斯大学埃尔帕索分校）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；Rochester Institute of Technology（罗切斯特理工学院）
Multi-object tracking and segmentation (MOTS) is a critical task for autonomous driving applications. The existing MOTS studies face two critical challenges: 1) the published datasets inadequately capture the real-world complexity for network training to address various driving settings; 2) the working pipeline annotation tool is under-studied in the literature to improve the quality of MOTS learning examples. In this work, we introduce the DG-Labeler and DGL-MOTS dataset to facilitate the training data annotation for the MOTS task and accordingly improve network training accuracy and efficien…
CORP: A Multi-Modal Dataset for Campus-Oriented Roadside Perception Tasks
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究院）；University of Science and Technology of China（中国科学技术大学）
Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tsinghua University（清华大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this work, we propose to learn such cross-category skills via Generalizable and Actionable Parts (GAParts). By identifying and defining 9 GAPart classes (lids, handles, etc.) in 27 object categories, we construct a large-scale part-centric interactive dataset, GAPartNet, where we provide rich, part-level annotations (semantics, poses) for 8,489 part instances on 1,166 objects. Based on GAPartNet, we investigate three cross-c…
Holistically understanding an object and its 3D movable parts through visual perception models is essential for enabling an autonomous agent to interact with the world. For autonomous driving, the dynamics and states of vehicle parts such as doors, the trunk, and the bonnet can provide meaningful semantic information and interaction states, which are essential to ensuring the safety of the self-driving vehicle. Existing visual perception models mainly focus on coarse parsing such as object bounding box detection or pose estimation and rarely tackle these situations. In this paper, we address t…
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Renault SAS（雷诺股份有限公司）
In this paper, a multi-modal 360$^{\circ}$ framework for 3D object detection and tracking for autonomous vehicles is presented. The process is divided into four main stages. First, images are fed into a CNN network to obtain instance segmentation of the surrounding road participants. Second, LiDAR-to-image association is performed for the estimated mask proposals. Then, the isolated points of every object are processed by a PointNet ensemble to compute their corresponding 3D bounding boxes and poses. Lastly, a tracking stage based on Unscented Kalman Filter is used to track the agents along ti…
Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Wuppertal（伍珀塔尔大学）；Interdisciplinary Center for Machine Learning and Data Analytics（机器学习与数据分析跨学科中心）
UniFS: Universal Few-shot Instance Perception with Point Representations
The University of Hong Kong（香港大学）；SenseTime Research and Tetras.AI（商汤科技研究院与Tetras.AI）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）
INSTINCT: Instance-Level Interaction Architecture for Query-Based Collaborative Perception
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Future Science and Engineering, Soochow University（苏州大学未来科学与工程学院）
UC San Diego（加利福尼亚大学圣地亚哥分校）；Toyota InfoTechnology Center（丰田信息技术中心）
Unsupervised learning for geometric perception (depth, optical flow, etc.) is of great interest to autonomous systems. Recent works on unsupervised learning have made considerable progress on perceiving geometry; however, they usually ignore the coherence of objects and perform poorly under scenarios with dark and noisy environments. In contrast, supervised learning algorithms, which are robust, require large labeled geometric dataset. This paper introduces SIGNet, a novel framework that provides robust geometry perception without requiring geometrically informative labels. Specifically, SIGNe…
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）
In the past few years, we have seen great progress in perception algorithms, particular through the use of deep learning. However, most existing approaches focus on a few categories of interest, which represent only a small fraction of the potential categories that robots need to handle in the real-world. Thus, identifying objects from unknown classes remains a challenging yet crucial task. In this paper, we develop a novel open-set instance segmentation algorithm for point clouds which can segment objects from both known and unknown classes in a holistic way. Our method uses a deep convolutio…
Driving requires interacting with road agents and predicting their future behaviour in order to navigate safely. We present FIERY: a probabilistic future prediction model in bird's-eye view from monocular cameras. Our model predicts future instance segmentation and motion of dynamic agents that can be transformed into non-parametric future trajectories. Our approach combines the perception, sensor fusion and prediction components of a traditional autonomous driving stack by estimating bird's-eye-view prediction directly from surround RGB monocular camera inputs. FIERY learns to model the inher…
CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception
Institute for AI Industry Research, Tsinghua University（人工智能产业研究所，清华大学）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；School of Vehicle and Mobility, Tsinghua University（车辆与移动技术学院，清华大学）；Baidu Inc.（百度公司）
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Southern University of Science and Technology（南方科技大学）
Compared with the visual grounding on 2D images, the natural-language-guided 3D object localization on point clouds is more challenging. In this paper, we propose a new model, named InstanceRefer, to achieve a superior 3D visual grounding through the grounding-by-matching strategy. In practice, our model first predicts the target category from the language descriptions using a simple language classification model. Then, based on the category, our model sifts out a small number of instance candidates (usually less than 20) from the panoptic segmentation of point clouds. Thus, the non-trivial 3D…
Due to the lack of depth information of images and poor detection accuracy in monocular 3D object detection, we proposed the instance depth for multi-scale monocular 3D object detection method. Firstly, to enhance the model's processing ability for different scale targets, a multi-scale perception module based on dilated convolution is designed, and the depth features containing multi-scale information are re-refined from both spatial and channel directions considering the inconsistency between feature maps of different scales. Firstly, we designed a multi-scale perception module based on dila…
Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
organization= College of Computer；Data Science , addressline= Fuzhou University , city= Fuzhou , country= China；organization= School of Mechanical Engineering；Automation , addressline= Fuzhou University , city= Fuzhou , country= China；organization= College of Physics；Information Engineering , addressline= Fuzhou University , city= Fuzhou , country= China
GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding
Northwestern Polytechnical University（西北工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Nanyang Technological University（南洋理工大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Hefei University of Technology（合肥工业大学）；Baidu, Inc.（百度公司）
In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-awa…
As the perception range of LiDAR increases, LiDAR-based 3D object detection becomes a dominant task in the long-range perception task of autonomous driving. The mainstream 3D object detectors usually build dense feature maps in the network backbone and prediction head. However, the computational and spatial costs on the dense feature map are quadratic to the perception range, which makes them hardly scale up to the long-range setting. To enable efficient long-range LiDAR-based object detection, we build a fully sparse 3D object detector (FSD). The computational and spatial cost of FSD is rough…
Image-Conditioned Instance Prompt Network for Referring Remote Sensing Image Segmentation
School of Computer Science, Northwestern Polytechnical University, Xi'an, China（西北工业大学计算机科学学院，西安，中国）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China（复旦大学智能机器人与先进制造学院，上海，中国）；Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen, China（西北工业大学深圳研究院，深圳，中国）
