发表机构
Bradley Department of Electrical and Computer Engineering, Virginia Tech; Virginia Tech Transportation Institute; Sanghani Center for Artificial Intelligence and Data Analytics（弗吉尼亚理工大学布拉德利电气与计算机工程系; 弗吉尼亚理工大学交通研究所; 桑哈尼人工智能与数据分析中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对自动驾驶中多模态大语言模型，提出D3VL框架，整合2D和3D时间序列数据，回答交通场景相关问题，在KITTI问答数据集上性能提升11%，并引入Waymo QA数据集扩展以评估模型在多样驾驶条件下处理3D和时间序列数据的能力。
AI 中文摘要
多模态大语言模型（MLLMs）的进展推动了自动驾驶端到端MLLMs的发展，但目前主要集中在使用2D图像和视频。本文考虑使用3D传感器（尤其是激光雷达和立体摄像头）的MLLM有效性。激光雷达集成面临数据稀疏等挑战，摄像头与激光雷达数据融合也少见。本文提出D3VL框架，能在单一架构中整合2D和3D时间序列数据，旨在回答交通场景理解和安全问题。在KITTI问答数据集上比基线方法提高11%，还引入Waymo QA数据集扩展。
英文摘要
Recent advances in Multimodal Large Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA dataset extension, which assesses models' capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: https://automotivesafety-lvlm.github.io
CommentsAccepted to IEEE IV 2026
