Deep neural networks (DNNs) remain challenged by distribution shifts in complex open-world domains like automated driving (AD): Robustness against yet unknown novel objects (semantic shift) or styles like lighting conditions (covariate shift) cannot be guaranteed. Hence, reliable operation-time monitors for identification of out-of-training-data-distribution (OOD) scenarios are imperative. Current approaches for OOD classification are untested for complex domains like AD, are limited in the kinds of shifts they detect, or even require supervision with OOD samples. To prepare for unanticipated shifts, we instead establish a framework around a principled, unsupervised and model-agnostic method that unifies detection of semantic and covariate shifts: Find a full model of the training data's feature distribution, to then use its density at new points as in-distribution (ID) score. To implement this, we propose to combine Vision Foundation Models (VFMs) as feature extractors with density modeling techniques. Through a comprehensive benchmark of 4 VFMs with different backbone architectures and 5 density-modeling techniques against established baselines, we provide the first systematic evaluation of OOD classification capabilities of VFMs across diverse conditions. A comparison with state-of-the-art binary OOD classification methods reveals that VFM embeddings with density estimation outperform existing approaches in identifying OOD inputs. Additionally, we show that our method detects high-risk inputs likely to cause errors in downstream tasks, thereby improving overall performance. Overall, VFMs, when coupled with robust density modeling techniques, are promising to realize model-agnostic, unsupervised, reliable safety monitors in complex vision tasks
4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions
Technical University of Munich(慕尼黑技术大学);Reality Labs at Meta(Meta现实实验室);Microsoft Mixed Reality & AI Lab(微软混合现实与人工智能实验室);Karlsruhe University of Applied Sciences(卡尔斯鲁厄应用科学大学)
AutoLay: Benchmarking amodal layout estimation for autonomous driving
Robotics Research Center, KCIS(KCIS机器人研究中心);IIIT Hyderabad(海得拉巴国际信息技术学院);Mila - Quebec AI Institute, Montreal(蒙特利尔魁北克人工智能研究所);Université de Montréal(蒙特利尔大学)
MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perception
Beijing Academy of Artificial Intelligence(北京人工智能研究院);IQIYI(爱奇艺);Institute of Automation CAS(中国科学院自动化研究所);Samsung(三星);The University of Sydney(悉尼大学);National University of Singapore(新加坡国立大学);Institute of Information Engineering CAS(中国科学院信息工程研究所);Hefei University of Technology(合肥工业大学)
Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving
Donald Bren School of Information and Computer Sciences, University of California, Irvine(加州大学欧文分校唐纳德·布伦信息与计算机科学学院);School of Computing, National University of Singapore(新加坡国立大学计算学院);Nanyang Technological University(南洋理工大学);Shanghai AI Laboratory(上海人工智能实验室)
Monitoring and Diagnosability of Perception Systems
Massachusetts Institute of Technology(麻省理工学院)
感知是机器人和自主系统(例如自动驾驶车辆)高完整性应用的关键组成部分。在这些应用中,感知系统的故障可能会危及人类生命,广泛采用这些技术需要开发方法来保证和监测安全操作。尽管感知系统至关重要,但目前还没有正式的系统级监控方法。在这项工作中,我们提出了一个数学模型的感知系统的运行时监测和故障检测和识别。为了实现这一目标,我们绘制连接与文献中的多处理器系统的可诊断性,并将其推广到考虑随着时间的推移与异构的输出交互的模块。由此产生的时间诊断图(i)提供了一个框架,原因的一致性的感知输出-跨模块和随着时间的推移-从而使故障检测,(ii)使我们能够建立正式的保证最大数量的故障,可以唯一地确定在一个给定的感知系统,和(iii)使设计有效的算法进行故障识别。我们使用LGSVL自动驾驶模拟器和Apollo Auto autonomy软件堆栈在现实模拟中演示了我们的监控系统PerSyS,并表明PerSyS能够在具有挑战性的场景中检测故障(包括近年来导致自动驾驶汽车事故的场景),并且能够正确识别故障,同时需要最小的计算开销(单核CPU上<5 ms)。
Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective
SKLCCSE, the School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院软件安全技术与工程北京市重点实验室);the School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络空间科学与技术学院);Henan University of Science and Technology(河南科技大学);the School of Computing, National University of Singapore(新加坡国立大学计算学院);College of Computing & Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models
School of Computer Science, Wuhan University(武汉大学计算机学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Waytous;University of Technology Sydney(悉尼大学);Microsoft Research(微软研究院);IAIR, Xi’an Jiaotong University(西安交通大学IAIR);TikTok;AIR, Tsinghua University(清华大学AIR)
Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring
McGill University(麦吉尔大学);Mila - Quebec Artificial Intelligence Institute(魁北克人工智能研究所);University of Copenhagen(哥本哈根大学);Agriculture and Agri-food Canada(加拿大农业与农业食品部);Smithsonian Tropical Research Institute(史密松尼热带研究所);Biology Center, Czech Academy of Sciences(捷克科学院生物学中心);Maestria de Entomologia, University of Panama(巴拿马大学昆虫学系);Massachusetts Institute of Technology(麻省理工学院);Montréal Insectarium(蒙特利尔昆虫馆)
Integrity Monitoring of 3D Object Detection in Automated Driving Systems using Raw Activation Patterns and Spatial Filtering
WMG, University of Warwick(华威大学WMG学院);School of Electronics, Electrical Engineering and Computer Science (EEECS), Queen’s University of Belfast(贝尔法斯特女王大学电子、电气工程与计算机科学学院)
AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving
Texas A&M University(德州农工大学);University of Toronto(多伦多大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Maryland(马里兰大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);UNC Chapel Hill(北卡罗来纳大学教堂山分校)
LanEvil: Benchmarking the Robustness of Lane Detection to Environmental Illusions
Beihang University(北京航空航天大学);Data Space Research Institute of Hefei Comprehensive National Science Centre(合肥综合性国家科学中心数据空间研究院);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)
PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Sichuan Agricultural University(四川农业大学)
CHIRP dataset: towards long-term, individual-level, behavioral monitoring of bird populations in the wild
Centre for the Advanced Study of Collective Behaviour, University of Konstanz(康斯坦茨大学集体行为高级研究中心);Dept. of Collective Behavior, Max Planck Institute of Animal Behavior(马克斯·普朗克动物行为研究所集体行为系);Dept. of Biology, University of Konstanz(康斯坦茨大学生物学系);Dept. of Computer and Information Science, University of Konstanz(康斯坦茨大学计算机与信息科学系);Luondua Boreal Field Station(卢昂杜阿北方野外站);School of Biological and Medical Sciences, Oxford Brookes University(牛津布鲁克斯大学生物与医学科学学院);Dept. of Zoology, Stockholm University(斯德哥尔摩大学动物学系);Dept. of Ecology of Animal Societies, Max Planck Institute of Animal Behavior(马克斯·普朗克动物行为研究所动物社会生态学系)
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);Robert Bosch GmbH(罗伯特·博世有限公司);OpenDriveLab at Shanghai AI Lab(上海人工智能实验室OpenDriveLab);NVIDIA Research(英伟达研究院);Nanyang Technological University(南洋理工大学);University of Toronto(多伦多大学);Vector Institute(向量研究院);Stanford University(斯坦福大学)
A Computer Vision Pipeline for Individual-Level Behavior Analysis: Benchmarking on the Edinburgh Pig Dataset
Department of Animal Science, College of Agriculture and Life Sciences, Cornell University(动物科学系,农业与生命科学学院,康奈尔大学);University of Liège, Gembloux Agro-Bio Tech (ULìege-GxABT)(列日大学,吉尔伯特农业生物技术(ULiège-GxABT))