arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.10845cs.CV
收藏

CoVLA:用于自动驾驶的全面视觉-语言-动作数据集

CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving

Turing Inc.(图灵公司)

Hidehisa Arai, Keita Miwa, Kento Sasaki, Yu Yamaguchi, Kohei Watanabe, Shunsuke Aoki, Issei Yamamoto

AI总结:

针对自动驾驶缺乏大规模视觉-语言-动作数据的问题,提出CoVLA数据集,含80多小时真实驾驶视频及自动化标注流程,验证了VLA模型在端到端驾驶中的潜力。

摘要:

自动驾驶,尤其是应对复杂和意外场景时,需要复杂的推理和规划能力。尽管多模态大语言模型(MLLMs)为此提供了有前景的途径,但其应用在很大程度上局限于理解复杂的环境背景或生成高级驾驶指令,很少有研究将其应用扩展到端到端的路径规划。一个主要的研究瓶颈是缺乏包含视觉、语言和动作的大规模标注数据集。为解决这一问题,我们提出了CoVLA(全面视觉-语言-动作)数据集,这是一个包含超过80小时真实世界驾驶视频的广泛数据集。该数据集利用一种新颖的、基于自动化数据处理和描述生成流程的可扩展方法,生成准确的驾驶轨迹,并配以详细的驾驶环境和操作的天然语言描述。该方法利用原始的车载传感器数据,使其在规模和标注丰富度上超越现有数据集。利用CoVLA,我们研究了能够在各种驾驶场景中处理视觉、语言和动作的MLLMs的驾驶能力。我们的结果展示了我们的模型在生成连贯的语言和动作输出方面的强大能力,强调了视觉-语言-动作(VLA)模型在自动驾驶领域的潜力。该数据集通过为训练和评估VLA模型提供全面的平台,为稳健、可解释和数据驱动的自动驾驶系统建立了框架,有助于实现更安全、更可靠的自动驾驶车辆。该数据集为学术目的而发布。

英文摘要:

Autonomous driving, particularly navigating complex and unanticipated scenarios, demands sophisticated reasoning and planning capabilities. While Multi-modal Large Language Models (MLLMs) offer a promising avenue for this, their use has been largely confined to understanding complex environmental contexts or generating high-level driving commands, with few studies extending their application to end-to-end path planning. A major research bottleneck is the lack of large-scale annotated datasets encompassing vision, language, and action. To address this issue, we propose CoVLA (Comprehensive Vision-Language-Action) Dataset, an extensive dataset comprising real-world driving videos spanning more than 80 hours. This dataset leverages a novel, scalable approach based on automated data processing and a caption generation pipeline to generate accurate driving trajectories paired with detailed natural language descriptions of driving environments and maneuvers. This approach utilizes raw in-vehicle sensor data, allowing it to surpass existing datasets in scale and annotation richness. Using CoVLA, we investigate the driving capabilities of MLLMs that can handle vision, language, and action in a variety of driving scenarios. Our results illustrate the strong proficiency of our model in generating coherent language and action outputs, emphasizing the potential of Vision-Language-Action (VLA) models in the field of autonomous driving. This dataset establishes a framework for robust, interpretable, and data-driven autonomous driving systems by providing a comprehensive platform for training and evaluating VLA models, contributing to safer and more reliable self-driving vehicles. The dataset is released for academic purpose.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑