arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.01442cs.CVcs.AIcs.CLcs.LG
收藏

CLEVRER:用于视频再现和推理的CoLlision事件

CLEVRER: CoLlision Events for Video REpresentation and Reasoning

Harvard University(哈佛大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);DeepMind(深度思维);MIT BCS, CBMM, CSAIL(麻省理工学院脑与认知科学系、大脑、思维与机器中心、计算机科学与人工智能实验室)

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum

摘要:

从视频中推理时间和因果事件的能力是人类智能的核心。然而,大多数视频推理基准集中在从复杂的视觉和语言输入的模式识别,而不是因果结构。我们研究的互补问题,探索的时间和因果结构背后的视频对象具有简单的视觉外观。为此,我们介绍了视频再现和推理(CLEVRER),诊断视频数据集的系统评估的计算模型在广泛的推理任务的冲突事件。受人类偶然判断理论的启发,CLEVRER包括四种类型的问题:描述性(例如,“什么颜色”)、解释性(“什么是负责的”)、预测性(“接下来会发生什么”)和反事实(“如果”)。我们评估各种国家的最先进的视觉推理模型在我们的基准。虽然这些模型在基于感知的任务(描述性)上表现出色,但它们在因果任务(解释性,预测性和反事实)上表现不佳,这表明因果推理的原则性方法应该结合感知复杂视觉和语言输入的能力,并理解潜在的动态和因果关系。我们还研究了一个甲骨文模型,明确结合这些组件通过符号表示。

英文摘要:

The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑