arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.23922cs.CVcs.CL
收藏

ScaleLong:面向长视频理解的多时间尺度基准

ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding

M-A-P;ByteDance Inc.(字节跳动)
David Ma, Huaqing Yuan, Xingjian Wang, Qianbo Zang, Tianci Liu, Xinyang He, Yanbin Wei, Jiawei Guo, Ni Jiahui, Zhenzhu Yang, Meng Cao, Shanghaoran Quan, Yizhi L… 展开作者

David Ma, Huaqing Yuan, Xingjian Wang, Qianbo Zang, Tianci Liu, Xinyang He, Yanbin Wei, Jiawei Guo, Ni Jiahui, Zhenzhu Yang, Meng Cao, Shanghaoran Quan, Yizhi Li, Wangchunshu Zhou, Jiaheng Liu, Wenhao Huang, Ge Zhang, Shiwen Ni, Xiaojie Jin

AI总结:

本文提出 ScaleLong 基准,通过在同一视频内容中嵌入四个层次时间尺度的问题,首次解耦了多时间尺度因素,评估 23 个 MLLM 揭示了 U 型性能曲线,并发现增加视觉 token 容量能持续提升各尺度的推理能力。

摘要:

尽管长视频理解要求模型捕捉层次化的时间信息——从片段(秒)和镜头(数十秒)到事件(分钟)和故事(小时)——但现有基准要么忽略了这种多尺度设计,要么将特定尺度的问题分散在不同的视频中,导致无法在同一内容上直接比较模型在不同时间尺度上的性能。为了解决这一问题,我们引入了 ScaleLong,这是第一个通过将针对四个层次时间尺度——片段(秒)、镜头(数十秒)、事件(分钟)和故事(小时)——的问题全部嵌入到同一视频内容中来解耦这些因素的基准。这种内容内多时间尺度提问设计使得能够在相同视频上直接比较模型在不同时间尺度上的性能。ScaleLong 包含来自 5 个主类别和 36 个子类别的 269 个长视频(平均 86 分钟),并配有 4-8 个精心设计的问题,每个时间尺度至少包含一个问题。对 23 个 MLLM 的评估揭示了 U 型性能曲线,在最短和最长时间尺度上准确率较高,而在中间水平出现下降。此外,消融研究表明,增加视觉 token 容量能持续提升所有时间尺度的推理能力。ScaleLong 为提升 MLLM 在长视频理解方面的能力提供了一个细粒度的多时间尺度基准。代码和数据集可在 https://github.com/multimodal-art-projection/ScaleLong 获取。

英文摘要:

Although long-video understanding demands that models capture hierarchical temporal information -- from clip (seconds) and shot (tens of seconds) to event (minutes) and story (hours) -- existing benchmarks either neglect this multi-scale design or scatter scale-specific questions across different videos, preventing direct comparison of model performance across timescales on the same content. To address this, we introduce ScaleLong, the first benchmark to disentangle these factors by embedding questions targeting four hierarchical timescales -- clip (seconds), shot (tens of seconds), event (minutes), and story (hours) -- all within the same video content. This within-content multi-timescale questioning design enables direct comparison of model performance across timescales on identical videos. ScaleLong features 269 long videos (avg.\ 86\,min) from 5 main categories and 36 sub-categories, with 4--8 carefully designed questions, including at least one question for each timescale. Evaluating 23 MLLMs reveals a U-shaped performance curve, with higher accuracy at the shortest and longest timescales and a dip at intermediate levels. Furthermore, ablation studies show that increased visual token capacity consistently enhances reasoning across all timescales. ScaleLong offers a fine-grained, multi-timescale benchmark for advancing MLLM capabilities in long-video understanding. The code and dataset are available https://github.com/multimodal-art-projection/ScaleLong.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑