arXiv 2501.066802025-07-30cs.CVcs.AIcs.LGcs.RO

视觉语言模型在自动驾驶中行人行为与场景理解的应用

Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving

ECE Department（电子工程系）；Carnegie Mellon University（卡内基梅隆大学）；Computer Science Department（计算机科学系）；Columbia University（哥伦比亚大学）；Rotman School of Management（罗特曼管理学院）；University of Toronto（多伦多大学）；Department of Statistics（统计学系）；George Washington University（乔治华盛顿大学）；Department of Computer Science（计算机科学系）；San Francisco State University（旧金山州立大学）

Haoxiang Gao, Li Zhang, Yu Zhao, Zhou Yang, Jinghan Cao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言模型在自动驾驶行人交互场景理解与高效部署的缺口，提出将大规模视觉语言基础模型知识蒸馏至高效视觉网络的方法，结合多预训练模型集成，显著提升开放词汇感知与轨迹预测性能，助力自动驾驶端到端能力增强。

摘要：

视觉语言模型（VLM）已成为提升自动驾驶感知与决策能力的一种颇具前景的方法。目前在将VLM应用于理解涉及行人交互的复杂场景以及高效车载部署方面仍存在差距。本文提出一种知识蒸馏方法，将大规模视觉语言基础模型的知识迁移至高效视觉网络，并将其应用于行人行为预测与场景理解任务，在生成更多样、更全面的语义属性方面取得了良好效果。我们还利用多个预训练模型与集成技术提升模型性能。我们进一步检验了知识蒸馏后模型的有效性，结果显示其在开放词汇感知与轨迹预测任务中的指标有显著提升，有望增强自动驾驶的端到端性能。

英文摘要：

Vision-language models (VLMs) have become a promising approach to enhancing perception and decision-making in autonomous driving. The gap remains in applying VLMs to understand complex scenarios interacting with pedestrians and efficient vehicle deployment. In this paper, we propose a knowledge distillation method that transfers knowledge from large-scale vision-language foundation models to efficient vision networks, and we apply it to pedestrian behavior prediction and scene understanding tasks, achieving promising results in generating more diverse and comprehensive semantic attributes. We also utilize multiple pre-trained models and ensemble techniques to boost the model's performance. We further examined the effectiveness of the model after knowledge distillation; the results show significant metric improvements in open-vocabulary perception and trajectory prediction tasks, which can potentially enhance the end-to-end performance of autonomous driving.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。