arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08571cs.ROcs.CV
收藏

面向自动驾驶的可扩展离线评估指标

Scalable Offline Metrics for Autonomous Driving

College of Engineering, Boston University(波士顿大学工程学院)

Animikh Aich, Adwait Kulkarni, Eshed Ohn-Bar

AI总结:

本文通过大量实验揭示自动驾驶策略离线与在线评估相关性弱于预期,并提出基于认知不确定性的离线指标,将相关性提升超过13%,并在真实环境中验证了其有效性。

摘要:

对机器人系统(如自动驾驶车辆)中基于感知的规划模型进行真实世界评估,可以通过离线方式安全且低成本地开展,即在预先采集并带有真值标注的验证数据集上计算模型预测误差。然而,将离线模型性能外推到在线环境仍然是一项挑战。在这些在线环境中,看似微小的误差可能会不断累积,最终导致测试阶段的违规行为或碰撞事故。这一关系尚未得到充分研究,尤其是在多样化的闭环指标和复杂的城市驾驶操作场景下。在本工作中,我们通过在不同条件和指标下开展大量实验,重新审视这一在策略评估中被低估的问题。基于仿真分析,我们发现离线与在线环境之间的相关性比先前研究报告的结果更差,这对当前驾驶策略评估实践和指标的有效性提出了质疑。接下来,我们弥合了离线评估与在线评估之间的差距。我们研究了一种基于认知不确定性的离线指标,该指标旨在捕捉在闭环环境中可能引发错误的事件。与以往的离线指标相比,所提出的指标在相关性上实现了超过13%的提升。我们进一步在真实世界环境中验证了研究结果的泛化能力,超越了仿真环境的局限,并观察到更大的性能增益。

英文摘要:

Real-world evaluation of perception-based planning models for robotic systems, such as autonomous vehicles, can be safely and inexpensively conducted offline, i.e. by computing model prediction error over a pre-collected validation dataset with ground-truth annotations. However, extrapolating from offline model performance to online settings remains a challenge. In these settings, seemingly minor errors can compound and result in test-time infractions or collisions. This relationship is understudied, particularly across diverse closed-loop metrics and complex urban maneuvers. In this work, we revisit this undervalued question in policy evaluation through an extensive set of experiments across diverse conditions and metrics. Based on analysis in simulation, we find an even worse correlation between offline and online settings than reported by prior studies, casting doubts on the validity of current evaluation practices and metrics for driving policies. Next, we bridge the gap between offline and online evaluation. We investigate an offline metric based on epistemic uncertainty, which aims to capture events that are likely to cause errors in closed-loop settings. The resulting metric achieves over 13% improvement in correlation compared to previous offline metrics. We further validate the generalization of our findings beyond the simulation environment in real-world settings, where even greater gains are observed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑