arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16647cs.CVcs.AI
收藏

DriveGenVLM:面向基于视觉语言模型的自动驾驶的真实世界视频生成

DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving

Columbia University(哥伦比亚大学);Data Science Institute, Columbia University(哥伦比亚大学数据科学研究所)

Yongjie Fu, Anmol Jain, Xuan Di, Xu Chen, Zhaobin Mo

AI总结:

提出DriveGenVLM框架,基于DDPM生成真实驾驶视频并利用EILEV视觉语言模型进行场景解说,以增强自动驾驶的交通理解、导航与规划能力。

摘要:

自动驾驶技术的发展需要日益复杂的方法来理解和预测真实世界场景。视觉语言模型(VLMs)正作为具有影响自动驾驶巨大潜力的革命性工具而兴起。在本文中,我们提出了DriveGenVLM框架来生成驾驶视频并使用VLMs来理解它们。为此,我们采用了一个基于去噪扩散概率模型(DDPM)的视频生成框架,旨在预测真实世界视频序列。随后,我们通过使用一个名为Egocentric Videos上的高效上下文学习(EILEV)的预训练模型,探讨了我们生成的视频用于VLMs的适用性。扩散模型使用Waymo开放数据集进行训练,并使用Fréchet Video Distance(FVD)分数进行评估,以确保生成视频的质量和真实感。EILEV为这些生成的视频提供了相应的解说,这在自动驾驶领域可能是有益的。这些解说可以增强交通场景理解,辅助导航,并提升规划能力。在DriveGenVLM框架中将视频生成与VLMs相结合,代表了利用先进AI模型应对自动驾驶中复杂挑战的重大进步。

英文摘要:

The advancement of autonomous driving technologies necessitates increasingly sophisticated methods for understanding and predicting real-world scenarios. Vision language models (VLMs) are emerging as revolutionary tools with significant potential to influence autonomous driving. In this paper, we propose the DriveGenVLM framework to generate driving videos and use VLMs to understand them. To achieve this, we employ a video generation framework grounded in denoising diffusion probabilistic models (DDPM) aimed at predicting real-world video sequences. We then explore the adequacy of our generated videos for use in VLMs by employing a pre-trained model known as Efficient In-context Learning on Egocentric Videos (EILEV). The diffusion model is trained with the Waymo open dataset and evaluated using the Fréchet Video Distance (FVD) score to ensure the quality and realism of the generated videos. Corresponding narrations are provided by EILEV for these generated videos, which may be beneficial in the autonomous driving domain. These narrations can enhance traffic scene understanding, aid in navigation, and improve planning capabilities. The integration of video generation with VLMs in the DriveGenVLM framework represents a significant step forward in leveraging advanced AI models to address complex challenges in autonomous driving.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑