arXiv 2603.288882026-03-30cs.RO

面向自动驾驶的语义观察层：VLMs在低延迟异常检测中的预部署可行性研究

A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

The City College of New York, Grove School of Engineering（纽约市立学院格罗夫工程学院）；Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University（纽约大学坦登工程学院机械与航空航天工程系）；Department of Electrical and Computer Engineering, Tandon School of Engineering, New York University（纽约大学坦登工程学院电气与计算机工程系）

Kunal Runwal, Swaraj Gajare, Daniel Adejumo, Omkar Ankalkope, Siddhant Baroth, Aliasghar Arab

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种语义观察层，通过量化视觉-语言模型实现低延迟异常检测，验证了其在自动驾驶平台上的可行性。

摘要：

语义异常-依赖上下文的危险-是像素级检测器无法推理的，对自动驾驶安全构成重大威胁。我们提出一种语义观察层：一个以1-2Hz运行的量化视觉-语言模型（VLM），与主自动驾驶控制回路并行运行，监控语义边缘情况，并在检测到时触发安全接管。使用Nvidia Cosmos-Reason1-7B模型与NVFP4量化和FlashAttention2，我们实现了约500毫秒的推理时间，比未优化的FP16基线（无量化，标准PyTorch注意力）快约50倍，满足观察器的时间预算。我们对静态和视频条件进行了准确性、延迟和量化行为的基准测试，识别出NF4召回崩溃（10.6%）作为硬部署限制，并将危险分析映射到安全目标。结果确立了语义观察器架构在具身人工智能自动驾驶平台上的预部署可行性。

英文摘要：

Semantic anomalies-context-dependent hazards that pixel-level detectors cannot reason about-pose a critical safety risk in autonomous driving. We propose a \emph{semantic observer layer}: a quantized vision-language model (VLM) running at 1--2\,Hz alongside the primary AV control loop, monitoring for semantic edge cases, and triggering fail-safe handoffs when detected. Using Nvidia Cosmos-Reason1-7B with NVFP4 quantization and FlashAttention2, we achieve ~500 ms inference a ~50x speedup over the unoptimized FP16 baseline (no quantization, standard PyTorch attention) on the same hardware--satisfying the observer timing budget. We benchmark accuracy, latency, and quantization behavior in static and video conditions, identify NF4 recall collapse (10.6%) as a hard deployment constraint, and a hazard analysis mapping performance metrics to safety goals. The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。