arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.02522cs.CVcs.AI
收藏

Moment-Video: 诊断视频多模态大语言模型在瞬时视觉事件上的时间保真度

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

Shanghai Jiao Tong University(上海交通大学);Shandong University(山东大学);Southeast University(东南大学);Tencent Youtu Lab(腾讯优图实验室)

Xiaolin Liu, Yilun Zhu, Xiangyu Zhao, Xuehui Wang, Yan Li, Xin Li, Haoyu Cao, Xing Sun, Shaofeng Zhang, Xu Yang, Zhihang Zhong, Xue Yang

AI总结:

提出 Moment-Video 基准,通过瞬时视觉事件理解任务诊断视频 MLLMs 的时间保真度,发现最佳模型准确率仅 39.6%,多数开源模型低于 25%。

摘要:

视频多模态大语言模型(MLLMs)在通用和长视频理解方面取得了快速进展,但它们保留简短答案关键视觉证据的能力仍未得到充分探索。许多实际问题由瞬时视觉事件决定:可能仅持续几帧的局部化动作或状态转换。这种证据可能因稀疏帧采样而跳过、因视觉标记压缩而抑制,或因粗粒度时间聚合而稀释,导致语言端推理无法可靠恢复的失败。我们引入了 Moment-Video,一个通过瞬时视觉事件理解来诊断视频 MLLMs 时间保真度的基准。每个问题都基于局部化、视觉可观察且对采样敏感的事件,要求模型注意、计数、描述或推理瞬态证据,而非依赖持久对象、全局场景上下文或语言先验。Moment-Video 包含 1,000 个人工验证的视频问答对,涵盖 7 个领域和 25 个细分子类别,覆盖四种任务类型:时间发生、时间计数、动作描述和时间推理。我们在 Moment-Video 上评估了 33 个专有和开源 MLLMs。最佳模型 Seed-2.0-Pro 仅达到 39.6% 的整体准确率,而大多数开源模型低于 25%,揭示了瞬时视觉事件理解方面的巨大差距。诊断分析表明,更密集的帧采样改善了一些模型,但并未消除瓶颈,更长的视频带来了更强的时间定位挑战。这些发现表明,当前视频 MLLMs 仍然缺乏时间保真的表示来捕捉、保留和使用简短但决定性的视觉证据。

英文摘要:

Video multimodal large language models (MLLMs) have made rapid progress in general and long-form video understanding, yet their ability to capture and use brief, answer-critical visual evidence remains underexplored. Momentary actions or state transitions may determine an answer while occupying only a small fraction of the video. Performance on broad video understanding tasks alone does not establish whether models can reliably identify and use this temporally localized evidence. We introduce Moment-Video, a benchmark for evaluating temporal fidelity through momentary visual event understanding. Questions are constructed around visually observable events or event sequences, requiring models to notice, count, describe, or reason about transient evidence. The benchmark contains 1,000 human-verified video-QA pairs across seven domains and 25 subcategories, covering four tasks: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. We evaluate 34 proprietary and open-source MLLMs. Under the main evaluation settings, the best-performing model, Seed-2.0-Pro, achieves 39.6% overall accuracy, while most open-source models remain below 25%. Diagnostic analyses show that denser frame sampling improves some models, but substantial errors persist even when sampled frames cover all annotated necessary event intervals in an audited subset. Performance also declines on longer videos, suggesting additional temporal-localization challenges. By exposing these limitations, Moment-Video provides a diagnostic benchmark to guide the evaluation and development of video MLLMs that more reliably understand brief, answer-critical visual events.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑