arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00959cs.AIcs.RO
收藏

将世界标记化为对象级知识以应对自动驾驶中的长尾事件

Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

NVIDIA(英伟达);UC Berkeley(加州大学伯克利分校);University of Southern California(南加州大学);Stanford University(斯坦福大学)

Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, Marco Pavone

AI总结:

本文提出TOKEN多模态大语言模型,通过将世界标记化为对象级知识并优化表示与推理对齐,有效缓解长尾场景下的数据稀缺问题,显著提升自动驾驶规划性能。

摘要:

自动驾驶行业正日益采用从感官输入进行端到端学习的方法,以尽量减少系统设计中的人为偏差。然而,传统的端到端驾驶模型由于训练分布中存在罕见或未见过的输入,容易受到长尾事件的影响。为了解决这一问题,我们提出了TOKEN,这是一种新颖的多模态大语言模型(MM-LLM),它将世界标记化为对象级知识,从而更好地利用大语言模型的推理能力来增强自动驾驶车辆在长尾场景中的规划能力。TOKEN通过利用传统的端到端驾驶模型生成紧凑且语义丰富的场景表示,有效缓解了数据稀缺和标记化效率低下的问题,并通过刻意的表示对齐和推理对齐训练阶段对这些表示进行优化,以使其与大语言模型规划兼容。我们的结果表明,TOKEN在接地、推理和规划能力方面表现出色,在长尾场景中优于现有框架,轨迹L2误差降低了27%,碰撞率降低了39%。此外,我们的工作强调了表示对齐和结构化推理在激发多模态大语言模型的常识推理能力以实现有效规划方面的重要性。

英文摘要:

The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs to minimize human biases in system design. Traditional end-to-end driving models, however, suffer from long-tail events due to rare or unseen inputs within their training distributions. To address this, we propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM's reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios. TOKEN effectively alleviates data scarcity and inefficient tokenization by leveraging a traditional end-to-end driving model to produce condensed and semantically enriched representations of the scene, which are optimized for LLM planning compatibility through deliberate representation and reasoning alignment training stages. Our results demonstrate that TOKEN excels in grounding, reasoning, and planning capabilities, outperforming existing frameworks with a 27% reduction in trajectory L2 error and a 39% decrease in collision rates in long-tail scenarios. Additionally, our work highlights the importance of representation alignment and structured reasoning in sparking the common-sense reasoning capabilities of MM-LLMs for effective planning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑