FIBER:填空作为一种可扩展的视频理解评估框架
FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework
摘要:
我们提出了填空作为视频理解评估框架,并引入了Fiber -一个由28,000个视频和描述组成的新数据集,以支持该评估框架。填空设置通过要求模型在给定视频和周围文本的情况下预测视频标题中的掩码名词短语来测试模型对视频的理解。FIBER基准测试不具有当前最先进的语言信息视频理解任务的弱点,即:(1)使用多项选择题的视频问答,其中模型表现相对较好,因为它们利用了任务制定中的语言偏见,从而使我们的框架对当前最先进的系统具有挑战性;以及(2)视频描述,其依赖于开放式评估框架,该框架通常是不准确的,因为如果系统回答在形式上与地面实况不同,则系统回答可能被认为是不正确的。FIBER数据集和我们的代码可以在https://lit.eecs.umich.edu/fiber/上找到。
英文摘要:
We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER -- a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-blanks setting tests a model's understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER benchmark does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation, thus making our framework challenging for the current state-of-the-art systems to solve; and (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https://lit.eecs.umich.edu/fiber/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。