MovieRecapsQA: 一种多模态开放式视频问答基准
MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark
AI总结：
本文提出MovieRecapsQA基准，通过电影回顾视频生成8200个开放式问题及事实，构建无参考评估指标，评估多模态能力，发现视觉感知是主要瓶颈。
摘要：
理解现实世界视频如电影需要整合视觉和对话线索。然而，现有视频问答基准在捕捉这种多模态推理方面存在困难，鉴于自由形式答案的评估难度，大多只能采用简单的多项选择题。我们引入了一个新的开放性多模态视频问答基准，MovieRecapsQA，该基准利用电影回顾视频创建，这是一种独特的YouTube内容类型，通过语音描述电影中的关键片段来总结电影（回顾视频）。从60个回顾视频的转录语音描述（回顾摘要）中，我们生成约8.2K个问题及每个答案所需的必要“事实”；前者有助于创建需要多模态推理的问题，后者允许构建一个可应用于开放性回答的无参考评估指标。据我们所知，这是首个无参考的开放性视频问答基准。该基准允许每个问题在不同的输入视频设置下进行评估：给定（a）完整长度的电影，（b）完整的（约11分钟）回顾视频（仅视觉），（c）约14分钟的对齐电影场景，即与问题相关的电影场景，以及（d）约1.2分钟的对齐回顾视频场景。在所有情况下，任何相关电影对话的文本都会提供。每个问题根据回答所需的模态进行分类——视觉、对话或两者，这使能够对多模态能力进行细粒度评估。我们基准测试（设置（d））七种最先进的MLLMs，并发现（i）只有我们的无参考指标能产生有意义的人类对齐模型分离；（ii）以视觉为中心的问题在所有模型中得分最低；（iii）去除视觉输入往往会提高模型的事实性；（iv）主要瓶颈是视觉感知，而不是视觉推理。
英文摘要：
Understanding real-world videos such as movies requires integrating visual and dialogue cues. Yet existing VideoQA benchmarks struggle to capture this multimodal reasoning and, given the difficulty of evaluating free-form answers, largely resort to simple multiple choice questions. We introduce a novel open-ended multimodal VideoQA benchmark, MovieRecapsQA, created using movie recap videos -- a distinctive type of YouTube content that summarizes a film via a voiceover description of key clips from the movie (recap video). From the transcribed voiceover (recap summary) of 60 recap videos, we generate $\approx$8.2K questions along with the necessary ``facts'' expected in each answer; the former facilitates the creation of questions that require mutimodal reasoning and the latter allow the construction of a reference-free evaluation metric that can be applied to open-ended responses. To our knowledge, this is the first reference-free open-ended VideoQA benchmark. The benchmark allows each question to be evaluated in different input video settings: given (a) the full-length movie, (b) the full ($\approx$11 min) recap video (visual only), (c) $\approx$14 min of aligned movie scenes, i.e, movie scenes relevant to the question, and (d) $\approx$1.2 min of aligned recap video scenes. In all cases, the text of any associated movie dialogue is provided. Each question is categorized by the modality required to answer it -- visual, dialogue, or both -- enabling fine-grained evaluation of multimodal capabilities. We benchmark (setting (d)) seven state-of-the-art MLLMs and find that (i) only our reference-free metric produces meaningful human-aligned model separation; (ii) vision-centric questions yield the lowest scores across all models; (iii) removing visual input often \textit{improves} model factuality; and (iv) the primary bottleneck is visual perception, not visual reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
