当语言与视觉交汇于道路安全：利用多模态大模型开展基于视频的交通事故分析
When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis
AI总结：
针对现有交通视频事故分析需大量人工后处理的问题，提出集成MLLM智能体的SeeUnsafe框架，采用严重度聚合策略与多模态提示，结合新指标IMS，在相关数据集上验证了其事故分类与视觉定位的有效性。
摘要：
全天候24/7/365运行的交通视频数量不断增加，极大提升了交通事故的时空覆盖潜力，将助力改善交通安全。然而，按照24/7/365的工作模式分析数百乃至数千个交通摄像头的拍摄素材仍是一项极具挑战性的任务——当前基于视觉的方法主要聚焦于提取车辆轨迹或单个目标检测等原始信息，需要耗费大量人力进行后处理才能得出可落地的洞察。我们提出SeeUnsafe这一全新框架，它集成了多模态大语言模型（Multimodal Large Language Model, MLLM）智能体，将基于视频的交通事故分析从传统的“提取-解释”工作流转变为更具交互性的对话式方法。这一转变通过自动化视频分类、视觉定位等复杂任务大幅提升了处理吞吐量，同时通过支持对多样交通场景和用户自定义查询的无缝调整增强了适配性。我们的框架采用基于严重程度的聚合策略来处理不同时长的视频，并使用一种新颖的多模态提示生成结构化响应以供审核评估，同时实现细粒度视觉定位。我们提出了信息匹配分数（Information Matching Score, IMS），这是一种基于MLLM的新型指标，用于对齐结构化响应与真实标注。我们在Toyota Woven Traffic Safety数据集上开展了大量实验，结果表明SeeUnsafe借助现成的MLLM可有效完成事故感知型视频分类与视觉定位任务。源代码将发布于https://github.com/ai4ce/SeeUnsafe。
英文摘要：
The increasing availability of traffic videos functioning on a 24/7/365 time scale has the great potential of increasing the spatio-temporal coverage of traffic accidents, which will help improve traffic safety. However, analyzing footage from hundreds, if not thousands, of traffic cameras in a 24/7/365 working protocol remains an extremely challenging task, as current vision-based approaches primarily focus on extracting raw information, such as vehicle trajectories or individual object detection, but require laborious post-processing to derive actionable insights. We propose SeeUnsafe, a new framework that integrates Multimodal Large Language Model (MLLM) agents to transform video-based traffic accident analysis from a traditional extraction-then-explanation workflow to a more interactive, conversational approach. This shift significantly enhances processing throughput by automating complex tasks like video classification and visual grounding, while improving adaptability by enabling seamless adjustments to diverse traffic scenarios and user-defined queries. Our framework employs a severity-based aggregation strategy to handle videos of various lengths and a novel multimodal prompt to generate structured responses for review and evaluation and enable fine-grained visual grounding. We introduce IMS (Information Matching Score), a new MLLM-based metric for aligning structured responses with ground truth. We conduct extensive experiments on the Toyota Woven Traffic Safety dataset, demonstrating that SeeUnsafe effectively performs accident-aware video classification and visual grounding by leveraging off-the-shelf MLLMs. Source code will be available at \url{https://github.com/ai4ce/SeeUnsafe}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
