TrafficVLM：一种用于交通视频描述的可控视觉语言模型
TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning
AI总结：
本文提出TrafficVLM，一种面向车辆自车视角的多模态密集视频描述模型，通过时空分层建模、条件生成控制和多任务微调，实现对车辆与行人的长篇幅细粒度描述，并在AI City Challenge 2024 Track 2中取得第三名。
摘要：
由于对高效可靠的城市监控系统需求日益增长，交通视频描述与分析近来受到广泛关注。现有方法大多只关注定位交通事件片段，严重缺乏与事件中所有感兴趣主体的行为和上下文相关的描述性细节。本文提出TrafficVLM，一种面向车辆自车视角的新型多模态密集视频描述模型。TrafficVLM在空间和时间两个维度上对不同分析层次的交通视频事件进行建模，并为车辆和行人在事件不同阶段生成长篇幅的细粒度描述。我们还为TrafficVLM提出一个条件组件来控制生成输出，并提出一种多任务微调范式以增强TrafficVLM的学习能力。实验表明，TrafficVLM在车辆视角和俯视摄像头视角下均表现良好。我们的解决方案在AI City Challenge 2024的Track 2中取得了优异成绩，在挑战赛排名中位列第三。我们的代码已公开于https://github.com/quangminhdinh/TrafficVLM。
英文摘要：
Traffic video description and analysis have received much attention recently due to the growing demand for efficient and reliable urban surveillance systems. Most existing methods only focus on locating traffic event segments, which severely lack descriptive details related to the behaviour and context of all the subjects of interest in the events. In this paper, we present TrafficVLM, a novel multi-modal dense video captioning model for vehicle ego camera view. TrafficVLM models traffic video events at different levels of analysis, both spatially and temporally, and generates long fine-grained descriptions for the vehicle and pedestrian at different phases of the event. We also propose a conditional component for TrafficVLM to control the generation outputs and a multi-task fine-tuning paradigm to enhance TrafficVLM's learning capability. Experiments show that TrafficVLM performs well on both vehicle and overhead camera views. Our solution achieved outstanding results in Track 2 of the AI City Challenge 2024, ranking us third in the challenge standings. Our code is publicly available at https://github.com/quangminhdinh/TrafficVLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
