arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.02687cs.CV
收藏

从视频中对物体和事件进行组合式物理推理

Compositional Physical Reasoning of Objects and Events from Videos

MIT-IBM Watson AI lab(MIT-IBM沃森人工智能实验室);New York University(纽约大学);Harvard University(哈佛大学);UIUC(伊利诺伊大学);UC Berkeley(加州大学伯克利分校);MIT(麻省理工学院);UMass Amherst(马萨诸塞大学阿姆赫斯特分校)

Zhenfang Chen, Shilong Dong, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, Chuang Gan

AI总结:

本文提出组合式物理推理(ComPhy)数据集和神经符号框架PCR,从视频中推断物体隐藏物理属性(如质量、电荷)并预测动态,实验表明现有模型能力有限,而PCR能有效推理并回答复杂问题。

摘要:

理解并推理自然世界中物体的物理属性是人工智能的一项基本挑战。虽然颜色和形状等一些属性可以直接观察,但质量、电荷等其他属性则隐藏在物体的视觉外观之下。本文针对从物体的运动和相互作用中推断这些隐藏物理属性,并基于推断出的物理属性预测相应动态这一独特挑战。我们首先引入了组合式物理推理(ComPhy)数据集。对于给定的一组物体,ComPhy包含它们在不同初始条件下运动和相互作用的有限视频。模型的评估基于其揭示组合式隐藏属性(如质量和电荷)的能力,并利用这些知识回答一系列问题。除了来自模拟器的合成视频外,我们还收集了一个真实世界数据集,以进一步测试不同模型的物理推理能力。我们在ComPhy上评估了最先进的视频推理模型,并揭示了它们在捕捉这些隐藏属性方面的有限能力,这导致了较差的性能。我们还提出了一种新颖的神经符号框架——物理概念推理器(PCR),它通过问答学习和推理可见及隐藏的物理属性。训练后,PCR展现出卓越的能力。它能够跨帧检测和关联物体,锚定可见和隐藏的物理属性,进行未来和反事实预测,并利用这些提取的表示来回答具有挑战性的问题。

英文摘要:

Understanding and reasoning about objects' physical properties in the natural world is a fundamental challenge in artificial intelligence. While some properties like colors and shapes can be directly observed, others, such as mass and electric charge, are hidden from the objects' visual appearance. This paper addresses the unique challenge of inferring these hidden physical properties from objects' motion and interactions and predicting corresponding dynamics based on the inferred physical properties. We first introduce the Compositional Physical Reasoning (ComPhy) dataset. For a given set of objects, ComPhy includes limited videos of them moving and interacting under different initial conditions. The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions. Besides the synthetic videos from simulators, we also collect a real-world dataset to show further test physical reasoning abilities of different models. We evaluate state-of-the-art video reasoning models on ComPhy and reveal their limited ability to capture these hidden properties, which leads to inferior performance. We also propose a novel neuro-symbolic framework, Physical Concept Reasoner (PCR), that learns and reasons about both visible and hidden physical properties from question answering. After training, PCR demonstrates remarkable capabilities. It can detect and associate objects across frames, ground visible and hidden physical properties, make future and counterfactual predictions, and utilize these extracted representations to answer challenging questions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑