arXiv 2606.022732026-06-01cs.CV

用于驾驶员监控系统的视觉语言模型：一个驾驶员活动描述数据集

Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Dataset

Fraunhofer IOSB（弗劳恩霍夫智能系统研究所）；Technische Hochschule Ingolstadt（图林根工业大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）

David J. Lerch, Sarath Mulugurthi, Manuel Martin, Frederik Diederichs, Rainer Stiefelhagen

备注：: Accepted at IEEE ITSC 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过创建Drive&Act数据集的详细自然语言版本，评估并微调视觉语言模型，以提升对驾驶员细微动作的识别能力，微调后的模型在跨数据集评估中表现更优。

摘要：

理解细微的驾驶员动作对于构建可靠的驾驶员监控系统至关重要。现有的视觉语言模型（VLM）在通用数据集上训练，难以识别驾驶员行为的细微差别。本文通过创建Drive&Act数据集的详细自然语言版本来解决这一限制。我们使用基于LLM的评分方法在新的基准上评估了三个VLM。它们在新基准上的表现表明，它们无法可靠地生成准确的细粒度驾驶员活动描述。基于标注的Drive&Act数据集，我们创建了一个新的Drive&Act描述数据集，其中包含细粒度描述，用于训练VLM理解驾驶员活动。在驾驶员监控数据集（DMD）上的跨数据集评估表明，在我们的新Drive&Act描述数据集上微调的VLM能够很好地泛化到DMD数据集中的动作。在我们的Drive&Act描述数据集上微调的VLM取得了76的ACCR分数，优于零样本VLM基线的66 ACCR分数。这些发现表明，用丰富描述的驾驶员动作来适应VLM可以显著提高其解释驾驶员行为的能力，同时也突显了需要更多样化的数据集以支持未来应用中更广泛泛化的需求。我们的Drive&Act描述数据集和代码将在GitHub上公开。

英文摘要：

Understanding subtle driver actions is essential for building reliable driver monitoring systems. Existing visionlanguage models (VLMs) are trained on general datasets and struggle to recognize fine distinctions in driver behaviors. This paper addresses this limitation by creating a detailed natural language version of the Drive&Act dataset. We evaluate three VLMs on our new benchmark using LLM-based scoring methods. Their performance on the new benchmark shows that they cannot reliably generate accurate fine-grained driver activity descriptions. Based on the labeled Drive&Act dataset we create a new Drive&Act description dataset containing finegrained descriptions to train VLMs on driver activity understanding. Cross dataset evaluation on the Driver Monitoring Dataset (DMD) shows that the VLM fine-tuned on our new Drive&Act description dataset generalizes well to actions in the DMD dataset. The VLM fine-tuned on our Drive&Act description dataset achieves an ACCR score of 76 outperforming the zero-shot VLM baseline with an ACCR score of 66. These findings demonstrate that adapting VLMs with richly described driver actions can significantly improve their ability to interpret driver behavior while also highlighting the need for more diverse datasets to support broader generalization in future applications. Our Drive&Act description dataset and code will be publicly available on GitHub.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。