arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.16554cs.LGcs.CV
收藏

使用语言模型生成分布外场景

Generating Out-Of-Distribution Scenarios Using Language Models

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);UMass Amherst(马萨诸塞大学阿默斯特分校);TRI(丰田研究所);MIT LIDS(麻省理工学院决策与控制系统实验室)

Erfan Aasi, Phat Nguyen, Shiva Sreeram, Guy Rosman, Sertac Karaman, Daniela Rus

AI总结:

本文提出利用大型语言模型构建分支树来生成多样化分布外驾驶场景,并在CARLA中仿真验证,同时引入OOD-ness指标和视觉语言模型导航评估,以提升自动驾驶安全测试的可靠性。

摘要:

由机器学习技术控制的自动驾驶车辆的部署,需要在多样化的真实世界环境中进行广泛测试,需要稳健地处理边缘情况和分布外场景,并需要进行全面的安全验证,以确保这些系统能够在不可预测的条件下安全有效地行驶。解决分布外(OOD)驾驶场景对于增强安全性至关重要,因为OOD场景有助于验证车辆自动驾驶栈中模型的可靠性。然而,生成OOD场景具有挑战性,因为它们在城市场景驾驶数据集中具有长尾分布且十分罕见。近年来,大型语言模型(LLMs)在自动驾驶领域展现出潜力,特别是在零样本泛化和常识推理能力方面。在本文中,我们利用这些LLM的优势,引入了一个用于生成多样化OOD驾驶场景的框架。我们的方法使用LLMs构建一棵分支树,其中每个分支代表一个独特的OOD场景。随后,这些场景在CARLA模拟器中使用一个自动化框架进行仿真,该框架将场景增强与相应的文本描述对齐。我们通过大量仿真评估了我们的框架,并通过衡量场景丰富度的多样性指标评估其性能。此外,我们引入了一个新的“OOD-ness”指标,用于量化生成的场景偏离典型城市场景驾驶条件的程度。进一步地,我们探索了现代视觉语言模型(VLMs)解释并安全通过这些仿真OOD场景的能力。我们的研究结果为语言模型在城市场景驾驶背景下处理OOD场景的可靠性提供了有价值的见解。

英文摘要:

The deployment of autonomous vehicles controlled by machine learning techniques requires extensive testing in diverse real-world environments, robust handling of edge cases and out-of-distribution scenarios, and comprehensive safety validation to ensure that these systems can navigate safely and effectively under unpredictable conditions. Addressing Out-Of-Distribution (OOD) driving scenarios is essential for enhancing safety, as OOD scenarios help validate the reliability of the models within the vehicle's autonomy stack. However, generating OOD scenarios is challenging due to their long-tailed distribution and rarity in urban driving dataset. Recently, Large Language Models (LLMs) have shown promise in autonomous driving, particularly for their zero-shot generalization and common-sense reasoning capabilities. In this paper, we leverage these LLM strengths to introduce a framework for generating diverse OOD driving scenarios. Our approach uses LLMs to construct a branching tree, where each branch represents a unique OOD scenario. These scenarios are then simulated in the CARLA simulator using an automated framework that aligns scene augmentation with the corresponding textual descriptions. We evaluate our framework through extensive simulations, and assess its performance via a diversity metric that measures the richness of the scenarios. Additionally, we introduce a new "OOD-ness" metric, which quantifies how much the generated scenarios deviate from typical urban driving conditions. Furthermore, we explore the capacity of modern Vision-Language Models (VLMs) to interpret and safely navigate through the simulated OOD scenarios. Our findings offer valuable insights into the reliability of language models in addressing OOD scenarios within the context of urban driving.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑