arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14129cs.CVcs.CLcs.MM
收藏

面向事件的长视频理解

Towards Event-oriented Long Video Understanding

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Baichuan Inc.(百川公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)

Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen

AI总结:

针对现有视频理解基准存在的捷径偏差问题,该研究提出面向事件的长视频理解基准Event-Bench,以及高性价比的视频指令合并方法VIM,实验证明其性能超越现有开源模型与GPT-4V。

摘要:

随着视频多模态大语言模型(MLLMs)的快速发展,众多基准被提出以评估其视频理解能力。然而,由于视频中缺乏丰富的事件,这些数据集可能存在捷径偏差:仅需少数帧即可推断出答案,无需观看完整视频。为解决该问题,我们提出Event-Bench,一个基于现有数据集与人工标注构建的面向事件的长视频理解基准。Event-Bench包含6项事件相关任务与2190个测试实例,可全面评估视频事件理解能力。此外,我们提出视频指令合并(VIM),这是一种高性价比方法,通过合并后的事件密集型视频指令增强视频MLLMs,解决人工标注的事件密集型数据稀缺的问题。大量实验表明,表现最佳的模型GPT-4o总体准确率达53.33%,显著优于最优开源模型41.42%。凭借有效的指令合成方法与自适应模型架构,VIM在Event-Bench上的表现超越了当前最优开源模型与GPT-4V。所有代码、数据与模型均在https://github.com/RUCAIBox/Event-Bench公开。

英文摘要:

With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑