使用CCTest与Carla Leaderboard对四款端到端AI自动驾驶系统的综合评估
A Comprehensive Evaluation of Four End-to-End AI Autopilots Using CCTest and the Carla Leaderboard
AI总结:
本研究将CCTest关键配置测试方法应用于四款端到端开源自动驾驶仪,对比其与模块化自动驾驶仪的失效异同,并发现CCTest结果与Carla Leaderboard评估存在显著差异,凸显了开发结合定性定量标准的客观评估方法的必要性。
摘要:
自动驾驶系统的端到端AI自动驾驶仪已成为传统模块化自动驾驶仪的一种有前景的替代方案,有望降低开发成本并减少因模块组合带来的缺陷。然而,它们也存在AI系统众所周知的问题,如非确定性、不可解释性和异常性。这自然引发了对其进行评估的问题,尤其是将其与现有模块化解决方案进行比较的问题。\n 本工作拓展了一项针对关键配置测试(critical configuration testing,CCTest)方法的研究,该方法此前已应用于四款开源模块化自动驾驶仪。该方法与其他方法的不同之处在于,它生成的测试用例可确保被测自动驾驶仪存在安全控制策略的可能性。由于仿真中观察到的任何事故都意味着被测自动驾驶仪的失效,因此这能够准确评估自动驾驶仪在关键场景下的安全驾驶能力。本文的贡献分为两方面。\n 首先,我们将CCTest方法应用于四款端到端开源自动驾驶仪——InterFuser、MILE、Transfuser和LMDrive,并将其测试结果与此前在Carla仿真环境中使用相同方法测试的四款模块化开源自动驾驶仪的结果进行比较。该比较明确了两类自动驾驶仪在关键配置下失效情况的异同点。\n 其次,我们将四款自动驾驶仪在Carla Leaderboard中的评估结果与CCTest结果进行比较。这一比较揭示了显著的差异,反映出测试用例生成标准和风险评估方法的不同。它凸显了开发结合定性与定量标准的客观评估方法的必要性。
英文摘要:
End-to-end AI autopilots for autonomous driving systems have emerged as a promising alternative to traditional modular autopilots, offering the potential to reduce development costs and mitigate defects arising from module composition. However, they suffer from the well-known problems of AI systems such as non-determinism, non-explainability, and anomalies. This naturally raises the question of their evaluation and, in particular, their comparison with existing modular solutions. This work extends a study of the critical configuration testing (CCTest) approach that has been applied to four open modular autopilots. This approach differs from others in that it generates test cases ensuring safe control policies are possible for the tested autopilots. This enables an accurate assessment of the ability to drive safely in critical situations, as any incident observed in the simulation involves the failure of a tested autopilot. The contribution of this paper is twofold. Firstly, we apply the CCTest approach to four end-to-end open autopilots, InterFuser, MILE, Transfuser, and LMDrive, and compare their test results with those of the four modular open autopilots previously tested with the same approach implemented in the Carla simulation environment. This comparison identifies both differences and similarities in the failures of the two autopilot types in critical configurations. Secondly, we compare the evaluations of the four autopilots carried out in the Carla Leaderboard with the CCTest results. This comparison reveals significant discrepancies, reflecting differences in test case generation criteria and risk assessment methods. It underlines the need to work towards the development of objective assessment methods combining qualitative and quantitative criteria.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。