Variants of accuracy and precision are the gold-standard by which the computer vision community measures progress of perception algorithms. One reason for the ubiquity of these metrics is that they are largely task-agnostic; we in general seek to detect zero false negatives or positives. The downside of these metrics is that, at worst, they penalize all incorrect detections equally without conditioning on the task or scene, and at best, heuristics need to be chosen to ensure that different mistakes count differently. In this paper, we propose a principled metric for 3D object detection specifically for the task of self-driving. The core idea behind our metric is to isolate the task of object detection and measure the impact the produced detections would induce on the downstream task of driving. Without hand-designing it to, we find that our metric penalizes many of the mistakes that other metrics penalize by design. In addition, our metric downweighs detections based on additional factors such as distance from a detection to the ego car and the speed of the detection in intuitive ways that other detection metrics do not. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time. Our project page including an evaluation server can be found at https://nv-tlabs.github.io/detection-relevance.
ShanghaiTech University（上海科技大学）；Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Kentucky（肯塔基大学）；University of Maryland, College Park（马里兰大学帕克分校）
Shanghai Jiao Tong University（上海交通大学）；University of Southern California（南加州大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vision-centric joint perception and prediction (PnP) has become an emerging trend in autonomous driving research. It predicts the future states of the traffic participants in the surrounding environment from raw RGB images. However, it is still a critical challenge to synchronize features obtained at multiple camera views and timestamps due to inevitable geometric distortions and further exploit those spatial-temporal features. To address this issue, we propose a temporal bird's-eye-view pyramid transformer (TBP-Former) for vision-centric PnP, which includes two novel designs. First, a pose-sy…
The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Human-centric perceptions include a variety of vision tasks, which have widespread industrial applications, including surveillance, autonomous driving, and the metaverse. It is desirable to have a general pretrain model for versatile human-centric downstream tasks. This paper forges ahead along this path from the aspects of both benchmark and pretraining methods. Specifically, we propose a \textbf{HumanBench} based on existing datasets to comprehensively evaluate on the common ground the generalization abilities of different pretraining methods on 19 datasets from 6 diverse downstream tasks, i…
TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving
Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；FusionBrain Lab（FusionBrain实验室）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；HSE University（高等经济大学）；Innopolis University（因诺波利斯大学）；NUST MISIS（国立研究型技术大学莫斯科国立钢铁合金学院）
In this paper, we present BEVerse, a unified framework for 3D perception and prediction based on multi-camera systems. Unlike existing studies focusing on the improvement of single-task approaches, BEVerse features in producing spatio-temporal Birds-Eye-View (BEV) representations from multi-camera videos and jointly reasoning about multiple tasks for vision-centric autonomous driving. Specifically, BEVerse first performs shared feature extraction and lifting to generate 4D BEV representations from multi-timestamp and multi-view images. After the ego-motion alignment, the spatio-temporal encode…
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations suc…
Uber ATG（Uber 先进技术集团）；University of Toronto（多伦多大学）
In this paper we propose a novel end-to-end learnable network that performs joint perception, prediction and motion planning for self-driving vehicles and produces interpretable intermediate representations. Unlike existing neural motion planners, our motion planning costs are consistent with our perception and prediction estimates. This is achieved by a novel differentiable semantic occupancy representation that is explicitly used as cost by the motion planning process. Our network is learned end-to-end from human demonstrations. The experiments in a large-scale manual-driving dataset and clo…
EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving
Professorship of Autonomous Vehicle Systems, Technical University of Munich, Munich, Germany（自动驾驶车辆系统教授职位，慕尼黑技术大学，德国慕尼黑）；Bayerische Motoren Werke AG, Munich, Germany（巴伐利亚发动机有限公司，德国慕尼黑）；Data Analytics and Machine Learning Group, Technical University of Munich, Munich, Germany（数据分析与机器学习小组，慕尼黑技术大学，德国慕尼黑）
In recent years, vision-centric perception has flourished in various autonomous driving tasks, including 3D detection, semantic map construction, motion forecasting, and depth estimation. Nevertheless, the latency of vision-centric approaches is too high for practical deployment (e.g., most camera-based 3D detectors have a runtime greater than 300ms). To bridge the gap between ideal research and real-world applications, it is necessary to quantify the trade-off between performance and efficiency. Traditionally, autonomous-driving perception benchmarks perform the offline evaluation, neglecting…
UniHCP: A Unified Model for Human-Centric Perceptions
The University of Sydney（悉尼大学）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Human-centric perception plays a vital role in vision and graphics. But their data annotations are prohibitively expensive. Therefore, it is desirable to have a versatile pre-train model that serves as a foundation for data-efficient downstream tasks transfer. To this end, we propose the Human-Centric Multi-Modal Contrastive Learning framework HCMoCo that leverages the multi-modal nature of human data (e.g. RGB, depth, 2D keypoints) for effective representation learning. The objective comes with two main challenges: dense pre-train for multi-modality data, efficient usage of sparse human prior…
Human-Centric Perception for Child Sexual Abuse Imagery
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）；Federal Police of Brazil（巴西联邦警察局）；Universidade Estadual de Campinas (UNICAMP)（坎皮纳斯州立大学）；University of Sheffield（谢菲尔德大学）
Scale-Aware Pre-Training for Human-Centric Visual Perception: Enabling Lightweight and Generalizable Models
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（深圳先进研究院，电子科学与技术大学）；Center for Future Media, University of Electronic Science and Technology of China（未来媒体中心，电子科学与技术大学）；TongJi University（同济大学）
Improving Vision-language Models with Perception-centric Process Reward Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Bytedance（字节跳动）；University of California, San Diego（加州大学圣地亚哥分校）；The Hong Kong University of Science and Technology（香港科技大学）
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Zhejiang University（浙江大学）；Baidu VIS（百度视觉技术部）；University of Science and Technology Beijing（北京科技大学）；South China University of Technology（华南理工大学）；Shanghai Jiao Tong University（上海交通大学）；Jilin University（吉林大学）
Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Fudan University（复旦大学）；Meituan（美团）；Shanghai Key Lab of Intell. Info. Processing（上海智能信息处理重点实验室）；Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）
Vehicle-centric Perception via Multimodal Structured Pre-training
Information Materials and Intelligent Sensing Laboratory of Anhui Province（安徽省信息材料与智能感知实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；the School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
University of Michigan（密歇根大学）；Robotics Institute（机器人研究所）
Visual perception tasks often require vast amounts of labelled data, including 3D poses and image space segmentation masks. The process of creating such training data sets can prove difficult or time-intensive to scale up to efficacy for general use. Consider the task of pose estimation for rigid objects. Deep neural network based approaches have shown good performance when trained on large, public datasets. However, adapting these networks for other novel objects, or fine-tuning existing models for different environments, requires significant time investment to generate newly labelled instanc…
ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大模型与智能治理重点实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Jiaotong University（北京交通大学）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；AresoX
