SafePLUG: 通过像素级洞察和时间锚定赋能多模态大语言模型以理解交通事故
SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding
AI总结：
本文提出SafePLUG框架，通过像素级理解和时间锚定提升多模态大语言模型在交通事故分析中的能力，引入新数据集并实现区域问答、像素分割、时间事件定位等任务的高性能表现。
摘要：
多模态大语言模型（MLLMs）在视觉-语言任务中取得了显著进展，并展示了在交通事故理解中的强大潜力。然而，现有领域内的MLLMs主要关注粗粒度的图像或视频层面理解，难以处理细粒度的视觉细节或局部场景组件，限制了其在复杂事故场景中的应用。为解决这些限制，我们提出了SafePLUG，一种新的框架，通过像素级理解和时间锚定赋能MLLMs进行全面的交通事故分析。SafePLUG支持任意形状的视觉提示进行区域感知问答，基于语言指令进行像素级分割，并能够识别交通事故场景中的时间锚定事件。为了推进MLLMs在交通事故理解中的发展，我们整理了一个包含多模态问答对的新数据集，聚焦于多样化的事故场景，具有详细的像素级标注和时间事件边界。实验结果表明，SafePLUG在多个任务上表现优异，包括基于区域的问答、像素级分割、时间事件定位和事故事件理解。这些能力为复杂交通场景的细粒度理解奠定了基础，有望提高驾驶安全并增强智能交通系统中的情境感知。代码、数据集和模型检查点将在https://zihaosheng.github.io/SafePLUG公开。
英文摘要：
Multimodal large language models (MLLMs) have achieved remarkable progress across a range of vision-language tasks and demonstrate strong potential for traffic accident understanding. However, existing MLLMs in this domain primarily focus on coarse-grained image-level or video-level comprehension and often struggle to handle fine-grained visual details or localized scene components, limiting their applicability in complex accident scenarios. To address these limitations, we propose SafePLUG, a novel framework that empowers MLLMs with both Pixel-Level Understanding and temporal Grounding for comprehensive traffic accident analysis. SafePLUG supports both arbitrary-shaped visual prompts for region-aware question answering and pixel-level segmentation based on language instructions, while also enabling the recognition of temporally anchored events in traffic accident scenarios. To advance the development of MLLMs for traffic accident understanding, we curate a new dataset containing multimodal question-answer pairs centered on diverse accident scenarios, with detailed pixel-level annotations and temporal event boundaries. Experimental results show that SafePLUG achieves strong performance on multiple tasks, including region-based question answering, pixel-level segmentation, temporal event localization, and accident event understanding. These capabilities lay a foundation for fine-grained understanding of complex traffic scenes, with the potential to improve driving safety and enhance situational awareness in smart transportation systems. The code, dataset, and model checkpoints will be made publicly available at: https://zihaosheng.github.io/SafePLUG
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
