arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.01533cs.CV
收藏

OmniDrive:用于自动驾驶且包含反事实推理的整体视觉-语言数据集

OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

NVIDIA(英伟达);The Hong Kong Polytechnic University(香港理工大学);Beijing Institute of Technology(北京理工大学)

Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, Jose M. Alvarez

AI总结:

针对VLMs从2D扩展到3D自动驾驶的挑战,提出含反事实推理的OmniDrive数据集,通过合成标注生成监督信号,探索两种框架评估对齐与感知的重要性,在DriveLM和nuScenes任务上获显著改进。

摘要:

视觉-语言模型(VLMs)的进步使自动驾驶领域对其强大推理能力的兴趣日益增长。然而,将这些能力从2D扩展到完整的3D理解对实际应用至关重要。为应对这一挑战,我们提出OmniDrive——一个整体视觉-语言数据集,通过反事实推理将智能体模型与3D驾驶任务对齐。该方法通过评估潜在场景及其结果来增强决策,类似人类驾驶员考虑替代行动。我们基于反事实的合成数据标注流程生成大规模、高质量数据集,提供更密集的监督信号,连接规划轨迹与基于语言的推理。此外,我们探索两种先进的OmniDrive-Agent框架(Omni-L和Omni-Q),评估视觉-语言对齐与3D感知的重要性,为设计有效LLM智能体提供关键见解。在DriveLM问答基准和nuScenes开环规划任务上的显著改进证明了我们数据集和方法的有效性。

英文摘要:

The advances in vision-language models (VLMs) have led to a growing interest in autonomous driving to leverage their strong reasoning capabilities. However, extending these capabilities from 2D to full 3D understanding is crucial for real-world applications. To address this challenge, we propose OmniDrive, a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning. This approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions. Our counterfactual-based synthetic data annotation process generates large-scale, high-quality datasets, providing denser supervision signals that bridge planning trajectories and language-based reasoning. Futher, we explore two advanced OmniDrive-Agent frameworks, namely Omni-L and Omni-Q, to assess the importance of vision-language alignment versus 3D perception, revealing critical insights into designing effective LLM-agents. Significant improvements on the DriveLM Q\&A benchmark and nuScenes open-loop planning demonstrate the effectiveness of our dataset and methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑