RAD：自动驾驶中利用视觉-语言模型进行元动作的检索增强决策
RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Driving
AI总结：
针对VLM空间感知不足与幻觉问题，提出RAD检索增强决策框架，结合三阶段RAG流程和NuScenes微调，提升自动驾驶元动作决策性能。
摘要：
准确理解和决策高层元动作对于确保可靠且安全的自动驾驶系统至关重要。尽管视觉-语言模型（VLM）已在各类自动驾驶任务中展现出显著潜力，但它们往往受到空间感知不足和幻觉等限制，从而降低其在复杂自动驾驶场景中的有效性。为应对这些挑战，我们提出检索增强决策（RAD）框架，这是一种旨在增强VLM在自动驾驶场景中可靠生成元动作能力的新型架构。RAD利用检索增强生成（RAG）流程，通过由嵌入流、检索流和生成流组成的三阶段过程动态提升决策准确性。此外，我们在一个源自NuScenes数据集的专门策划数据集上微调VLM，以增强其空间感知和鸟瞰图图像理解能力。在基于NuScenes构建的策划数据集上进行的大量实验评估表明，RAD在匹配准确率、F1分数以及自定义总分等关键评估指标上均优于基线方法，突显了其在改进自动驾驶任务元动作决策方面的有效性。
英文摘要：
Accurately understanding and deciding high-level meta-actions is essential for ensuring reliable and safe autonomous driving systems. While vision-language models (VLMs) have shown significant potential in various autonomous driving tasks, they often suffer from limitations such as inadequate spatial perception and hallucination, reducing their effectiveness in complex autonomous driving scenarios. To address these challenges, we propose a retrieval-augmented decision-making (RAD) framework, a novel architecture designed to enhance VLMs' capabilities to reliably generate meta-actions in autonomous driving scenes. RAD leverages a retrieval-augmented generation (RAG) pipeline to dynamically improve decision accuracy through a three-stage process consisting of the embedding flow, retrieving flow, and generating flow. Additionally, we fine-tune VLMs on a specifically curated dataset derived from the NuScenes dataset to enhance their spatial perception and bird's-eye view image comprehension capabilities. Extensive experimental evaluations on the curated NuScenes-based dataset demonstrate that RAD outperforms baseline methods across key evaluation metrics, including match accuracy, and F1 score, and self-defined overall score, highlighting its effectiveness in improving meta-action decision-making for autonomous driving tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
