一种通过最小视频对实现物理理解的捷径感知型视频问答基准
A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
AI总结：
针对现有视频问答基准易因表层捷径导致分数虚高的问题，本文提出含5.5万样本的MVP基准，通过最小变化视频对评估模型物理理解能力，发现当前最优开源模型性能远低于人类。
摘要：
现有用于评估视频语言模型时空理解与推理能力的基准，由于存在基于表层视觉或文本线索的捷径解，容易出现分数虚高的问题。本文通过引入Minimal Video Pairs（MVP，最小视频对）基准来缓解准确评估模型性能的难题，这是一种用于评估视频语言模型物理理解能力的简易捷径感知型视频问答基准。该基准包含5.5万个聚焦于物理世界理解的高质量多选视频问答样本。样本从9个视频数据源中筛选而来，涵盖第一人称自我视角与第三人称外视角视频、机器人交互数据以及认知科学直觉物理基准。为了缓解依赖表层视觉或文本线索与偏差的捷径解，MVP中的每个样本都有一个最小变化配对——一段视觉相似的视频，搭配完全相同的问题但答案相反。要正确回答问题，模型必须对最小变化对中的两个样本都给出正确答案；因此，仅依赖视觉或文本偏差的模型性能会低于随机水平。人类在MVP上的表现为92.9%，而表现最佳的开源前沿视频语言模型准确率为40.2%，随机水平为25%。
英文摘要：
Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair -- a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9\%, while the best open-source state-of-the-art video-language model achieves 40.2\% compared to random performance at 25\%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
