arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07488cs.CVcs.AIcs.RO
收藏

LMDrive:基于大型语言模型的闭环端到端驾驶

LMDrive: Closed-Loop End-to-End Driving with Large Language Models

CUHK(香港中文大学);SenseTime Research(商汤科技研究院);CPII under InnoHK(InnoHK旗下CPII);University of Toronto(多伦多大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, Hongsheng Li

AI总结:

本文提出LMDrive,首个利用大型语言模型实现闭环端到端自动驾驶的框架,整合多模态数据与自然语言指令,并发布数据集与基准以验证其有效性。

摘要:

尽管近年来自动驾驶领域取得了显著进展,但现代方法在遇到长尾意外事件和具有挑战性的城市场景时,仍然存在困难,并可能导致严重事故。一方面,大型语言模型(LLM)展现出了接近“通用人工智能”的令人印象深刻的推理能力。另一方面,以往的自动驾驶方法往往依赖于有限格式的输入(如传感器数据和导航路径点),限制了车辆理解语言信息和与人类交互的能力。为此,本文介绍了LMDrive,一种新颖的、语言引导的、端到端的闭环自动驾驶框架。LMDrive独特地处理并整合多模态传感器数据与自然语言指令,使得在现实指令环境中与人类和导航软件进行交互成为可能。为了促进基于语言的闭环自动驾驶的进一步研究,我们还公开了相应的数据集,该数据集包含约64K条指令跟随数据片段,以及LangAuto基准,该基准测试系统处理复杂指令和具有挑战性驾驶场景的能力。我们进行了广泛的闭环实验,以证明LMDrive的有效性。据我们所知,我们是首个利用LLM进行闭环端到端自动驾驶的工作。代码、模型和数据集可在https://github.com/opendilab/LMDrive找到。

英文摘要:

Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑