arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.05015cs.CV
收藏

Video-MME-v2:迈向综合视频理解基准的下一阶段

Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, Yongkang Xie, Xiawu Zheng, Xue Y… 展开作者

Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, Yongkang Xie, Xiawu Zheng, Xue Yang, Haoyu Cao, Yunsheng Wu, Ziwei Liu, Xing Sun, Caifeng Shan, Ran He

AI总结:

Video-MME-v2通过三级层次结构和非线性评估策略,评估视频理解的鲁棒性和准确性,揭示当前模型与人类专家间的差距及多模态推理瓶颈。

摘要:

随着视频理解的快速发展,现有基准日益饱和,导致排行榜分数与实际模型能力之间存在显著差距。为解决这一问题,我们引入Video-MME-v2,一个全面评估视频理解鲁棒性和准确性的基准。我们设计了递进三级层次结构,逐步增加视频理解的复杂性,从多点视觉信息聚合到时间动态建模,最终到复杂多模态推理。此外,我们提出了基于组的非线性评估策略,强制相关查询之间的一致性和多步骤推理的连贯性。它惩罚碎片化或猜测正确的答案,并只奖励由有效推理支持的答案。为了保证数据质量,Video-MME-v2通过严格控制的人工标注流程构建,包括12名标注员和50名独立评审员。依托3,300人小时和多达5轮的质量保证,Video-MME-v2旨在成为最权威的视频基准之一。大量实验揭示了当前最佳模型Gemini-3-Pro与人类专家之间的显著差距,并揭示了一个清晰的层次瓶颈,其中视觉信息聚合和时间建模的错误会传播到限制高层推理。我们进一步发现,基于思考的推理高度依赖文本提示,使用字幕可以提高性能,但在纯视觉设置中有时会降低性能。通过暴露这些限制,Video-MME-v2建立了一个对下一代视频MLLMs开发具有挑战性的新测试平台。

英文摘要:

With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a \textbf{progressive tri-level hierarchy} that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a \textbf{group-based non-linear evaluation} strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by \textbf{3,300 human-hours} and up to \textbf{5 rounds} of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑