现实检验:利用大型语言模型评估驾驶场景的真实性
Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models
AI总结:
本研究实证评估了大型语言模型在判断自动驾驶测试场景真实性方面的有效性与鲁棒性,发现GPT表现最优,且道路和天气条件影响模型鲁棒性。
摘要:
大型语言模型(LLMs)在文本生成、摘要和分类等任务中展现出巨大的潜力。鉴于这些模型在海量在线知识上进行训练,我们假设LLMs能够评估自动驾驶测试技术生成的驾驶场景是否真实,即是否与现实世界的驾驶条件一致。为验证这一假设,我们进行了一项实证评估,以检验LLMs在执行该任务时的有效性和鲁棒性。这一现实检验是迈向基于LLM的自动驾驶测试技术的重要一步。在我们的实证评估中,我们从\f{deepscenario}(一个开放的驾驶场景数据集)中选取了64个真实场景。接下来,通过引入微小改动,我们创建了512个额外的真实场景,形成总计576个场景的数据集。利用该数据集,我们评估了三个LLMs(\f{gpt}、\f{llama}和\f{mistral})在评估驾驶场景真实性方面的鲁棒性。结果表明:(1)总体而言,\f{gpt}在几乎所有场景、道路和天气条件下均持续展现出最高的鲁棒性,优于\f{llama}和\f{mistral};(2)\f{mistral}的表现持续最差;(3)\f{llama}在特定条件下取得了良好结果;(4)道路和天气条件确实影响LLMs的鲁棒性。
英文摘要:
Large Language Models (LLMs) are demonstrating outstanding potential for tasks such as text generation, summarization, and classification. Given that such models are trained on a humongous amount of online knowledge, we hypothesize that LLMs can assess whether driving scenarios generated by autonomous driving testing techniques are realistic, i.e., being aligned with real-world driving conditions. To test this hypothesis, we conducted an empirical evaluation to assess whether LLMs are effective and robust in performing the task. This reality check is an important step towards devising LLM-based autonomous driving testing techniques. For our empirical evaluation, we selected 64 realistic scenarios from \deepscenario--an open driving scenario dataset. Next, by introducing minor changes to them, we created 512 additional realistic scenarios, to form an overall dataset of 576 scenarios. With this dataset, we evaluated three LLMs (\gpt, \llama, and \mistral) to assess their robustness in assessing the realism of driving scenarios. Our results show that: (1) Overall, \gpt achieved the highest robustness compared to \llama and \mistral, consistently throughout almost all scenarios, roads, and weather conditions; (2) \mistral performed the worst consistently; (3) \llama achieved good results under certain conditions; and (4) roads and weather conditions do influence the robustness of the LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。