arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.21293cs.CVcs.AI
收藏

SimpleLLM4AD:一种面向自动驾驶的端到端视觉-语言模型与图视觉问答方法

SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving

IEIT Systems(IEIT系统公司)

Peiru Zheng, Yun Zhao, Zhan Gong, Hong Zhu, Shaohua Wu

AI总结:

SimpleLLM4AD提出将端到端自动驾驶分解为感知、预测、规划、行为四阶段,通过视觉-语言模型逐阶段推理图视觉问答(GVQA)中的VQA对,实现基于语言的端到端驾驶,并在复杂场景中取得有竞争力的性能。

摘要:

许多领域都能从大语言模型(LLMs)的快速发展中受益。随着LLMs支持越来越多的模态,端到端自动驾驶(e2eAD)是面临新机遇的典型领域之一。在此,我们利用视觉-语言模型(VLM)提出了一种名为SimpleLLM4AD的端到端自动驾驶方法。在我们的方法中,端到端自动驾驶任务被划分为四个阶段,即感知、预测、规划和行为。每个阶段由若干视觉问答(VQA)对组成,并且VQA对之间相互连接,构建出一个称为图视觉问答(Graph VQA,GVQA)的图。通过VLM在GVQA中逐阶段推理每个VQA对,我们的方法能够实现基于语言的端到端驾驶。在我们的方法中,采用视觉Transformer(ViT)模型处理nuScenes视觉数据,同时利用VLM对从视觉输入中提取的信息进行解释和推理。在感知阶段,系统识别并分类驾驶环境中的物体。预测阶段涉及预测这些物体的潜在运动。规划阶段利用收集到的信息制定驾驶策略,确保自动驾驶车辆的安全性和效率。最后,行为阶段将规划好的动作转换为车辆可执行的指令。我们的实验表明,SimpleLLM4AD在复杂驾驶场景中取得了具有竞争力的性能。

英文摘要:

Many fields could benefit from the rapid development of the large language models (LLMs). The end-to-end autonomous driving (e2eAD) is one of the typically fields facing new opportunities as the LLMs have supported more and more modalities. Here, by utilizing vision-language model (VLM), we proposed an e2eAD method called SimpleLLM4AD. In our method, the e2eAD task are divided into four stages, which are perception, prediction, planning, and behavior. Each stage consists of several visual question answering (VQA) pairs and VQA pairs interconnect with each other constructing a graph called Graph VQA (GVQA). By reasoning each VQA pair in the GVQA through VLM stage by stage, our method could achieve e2e driving with language. In our method, vision transformers (ViT) models are employed to process nuScenes visual data, while VLM are utilized to interpret and reason about the information extracted from the visual inputs. In the perception stage, the system identifies and classifies objects from the driving environment. The prediction stage involves forecasting the potential movements of these objects. The planning stage utilizes the gathered information to develop a driving strategy, ensuring the safety and efficiency of the autonomous vehicle. Finally, the behavior stage translates the planned actions into executable commands for the vehicle. Our experiments demonstrate that SimpleLLM4AD achieves competitive performance in complex driving scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑