Understanding and reasoning about physics is an important ability of intelligent agents. We develop the PHYRE benchmark for physical reasoning that contains a set of simple classical mechanics puzzles in a 2D physical environment. The benchmark is designed to encourage the development of learning algorithms that are sample-efficient and generalize well across puzzles. We test several modern learning algorithms on PHYRE and find that these algorithms fall short in solving the puzzles efficiently. We expect that PHYRE will encourage the development of novel sample-efficient agents that learn efficient but useful models of physics. For code and to play PHYRE for yourself, please visit https://player.phyre.ai.
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Key Laboratory of General Artificial Intelligence, BIGAI（北京智源人工智能研究院通用人工智能国家重点实验室）；Department of Automation, Tsinghua University（清华大学自动化系）
Physical reasoning requires forward prediction: the ability to forecast what will happen next given some initial world state. We study the performance of state-of-the-art forward-prediction models in the complex physical-reasoning tasks of the PHYRE benchmark. We do so by incorporating models that operate on object or pixel-based representations of the world into simple physical-reasoning agents. We find that forward-prediction models can improve physical-reasoning performance, particularly on complex tasks that involve many objects. However, we also find that these improvements are contingent…
Reasoning about the behaviour of physical objects is a key capability of agents operating in physical worlds. Humans are very experienced in physical reasoning while it remains a major challenge for AI. To facilitate research addressing this problem, several benchmarks have been proposed recently. However, these benchmarks do not enable us to measure an agent's granular physical reasoning capabilities when solving a complex reasoning task. In this paper, we propose a new benchmark for physical reasoning that allows us to test individual physical reasoning capabilities. Inspired by how humans a…
School of Computing, The Australian National University（澳大利亚国立大学计算学院）
Humans are well-versed in reasoning about the behaviors of physical objects and choosing actions accordingly to accomplish tasks, while it remains a major challenge for AI. To facilitate research addressing this problem, we propose a new testbed that requires an agent to reason about physical scenarios and take an action appropriately. Inspired by the physical knowledge acquired in infancy and the capabilities required for robots to operate in real-world environments, we identify 15 essential physical scenarios. We create a wide variety of distinct task templates, and we ensure all the task te…
A common approach to solving physical reasoning tasks is to train a value learner on example tasks. A limitation of such an approach is that it requires learning about object dynamics solely from reward values assigned to the final state of a rollout of the environment. This study aims to address this limitation by augmenting the reward value with self-supervised signals about object dynamics. Specifically, we train the model to characterize the similarity of two environment rollouts, jointly with predicting the outcome of the reasoning task. This similarity can be defined as a distance measur…
Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）
In order for AI to be safely deployed in real-world scenarios such as hospitals, schools, and the workplace, it must be able to robustly reason about the physical world. Fundamental to this reasoning is physical common sense: understanding the physical properties and affordances of available objects, how they can be manipulated, and how they interact with other objects. Physical commonsense reasoning is fundamentally a multi-sensory task, since physical properties are manifested through multiple modalities - two of them being vision and acoustics. Our paper takes a step towards real-world phys…
University of Bielefeld（比勒费尔德大学）；Ruhr University Bochum（波鸿鲁尔大学）；University of Waterloo（滑铁卢大学）；Vector Institute（矢量研究所）；University of Toronto（多伦多大学）；Stanford University（斯坦福大学）；Georgia Institute of Technology（佐治亚理工学院）
UCLA（加州大学洛杉矶分校）；Stanford University（斯坦福大学）；MIT BCS, CBMM, CSAIL（麻省理工学院脑与认知科学系、心智/大脑/机器研究中心、计算机科学与人工智能实验室）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies. Such composite structures could induce a rich set of semantic concepts and relations, thus playing an important role in the interpretation and organization of visual signals as well as for the generalization of visual perception and reasoning. However, existing visual reasoning benchmarks mostly focus on objects rather than parts. Visual reasoning based on the full part-whole hierarchy is much more challenging than object-cent…
MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）；Harvard University（哈佛大学）；MIT（麻省理工学院）；The University of Hong Kong（香港大学）
Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not directly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interacti…
Bielefeld University（比勒费尔德大学）；Indian Institute of Information Technology（印度信息技术学院）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Nvidia（英伟达）
We propose a new deep learning model for goal-driven tasks that require intuitive physical reasoning and intervention in the scene to achieve a desired end goal. Its modular structure is motivated by hypothesizing a sequence of intuitive steps that humans apply when trying to solve such a task. The model first predicts the path the target object would follow without intervention and the path the target object should follow in order to solve the task. Next, it predicts the desired path of the action object and generates the placement of the action object. All components of the model are trained…
The University of Hong Kong（香港大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Stanford University（斯坦福大学）；MIT BCS, CBMM, CSAIL（麻省理工学院脑与认知科学系、大脑、心智与机器中心、计算机科学与人工智能实验室）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）
We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, which are impractical to obtain in real life. In this paper, we present the Dynamic Concept Learner (DCL), a unified framework that grounds physical objects and events from video and language. DCL first adopts a trajectory extractor to track each object over time and to represent it as a latent, object-centric feature vector. Building upon this object-centric representation, DC…
Inducing Causal World Models in LLMs for Zero-Shot Physical Reasoning
Department of Electrical Engineering（电气工程系）；Indian Institute of Technology Bombay（印度理工学院孟买分校）；Department of Computer Science and Automation（计算机科学与自动化系）；Indian Institute of Science（印度科学研究所）；Department of Computer Science（计算机科学系）；San Francisco State University（旧金山州立大学）；University of Lagos（拉各斯大学）；Faculty of Mathematics, Informatics, and Mechanics（数学、信息学与力学学院）；University of Warsaw（华沙大学）
Scaling Physical Reasoning with the PHYSICS Dataset
Shanghai AI Laboratory（上海人工智能实验室）；Harbin Institute of Technology（哈尔滨工业大学）；Tsinghua University（清华大学）；CUHK（香港中文大学）；BUAA（北京航空航天大学）；UESTC（电子科技大学）；Soochow University（苏州大学）；USTC（中国科学技术大学）
Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tsinghua University（清华大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Is dynamics prediction indispensable for physical reasoning? If so, what kind of roles do the dynamics prediction modules play during the physical reasoning process? Most studies focus on designing dynamics prediction networks and treating physical reasoning as a downstream task without investigating the questions above, taking for granted that the designed dynamics prediction would undoubtedly help the reasoning process. In this work, we take a closer look at this assumption, exploring this fundamental hypothesis by comparing two learning mechanisms: Learning from Dynamics (LfD) and Learning …
Denoising Hamiltonian Network for Physical Reasoning
Massachusetts Institute of Technology（麻省理工学院）；Stanford University（斯坦福大学）；Harvard-Smithsonian Center for Astrophysics（哈佛-史密森尼天体物理中心）；Northeastern University（东北大学）
Neural Force Field: Few-shot Learning of Generalized Physical Reasoning
Institute for AI, Peking University（人工智能研究院，北京大学）；School of Psychological and Cognitive Sciences, Peking University（心理学与认知科学学院，北京大学）；School of EECS, Peking University（电子工程学院，北京大学）；School of Integrated Circuits, Peking University（集成电路学院，北京大学）；State Key Lab of General AI, Peking University（通用人工智能国家重点实验室，北京大学）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京行为与心理健康重点实验室，北京大学）
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Microsoft Research AI（微软研究院AI）；Carnegie Mellon University（卡内基梅隆大学）；Paul G. Allen School for Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we in…
Massachusetts Institute of Technology（麻省理工学院）；University of California, Berkeley（加州大学伯克利分校）
Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of…
Institute for Infocomm Research, A*STAR（A*STAR信息通信研究所）；National University of Singapore（新加坡国立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Centre for Frontier AI Research, A*STAR（A*STAR前沿人工智能研究中心）
Research in cognitive science has provided extensive evidence of human cognitive ability in performing physical reasoning of objects from noisy perceptual inputs. Such a cognitive ability is commonly known as intuitive physics. With advancements in deep learning, there is an increasing interest in building intelligent systems that are capable of performing physical reasoning from a given scene for the purpose of building better AI systems. As a result, many contemporary approaches in modelling intuitive physics for machine cognition have been inspired by literature from cognitive science. Desp…
Vision-language models lag human performance on physical dynamics and intent reasoning
East China Normal University（东华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；University of Agder（阿格德大学）
Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control
University of Michigan（密歇根大学）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Dalle Molle Institute for Artificial Intelligence Research (IDSIA)（达勒莫尔人工智能研究所）；Università della Svizzera italiana (USI)（瑞士意大利语区大学）；Scuola universitaria professionale della Svizzera italiana (SUPSI)（瑞士意大利语区应用科学与艺术大学）
Massachusetts Institute of Technology（麻省理工学院）；Center for Brains, Minds and Machines（脑、心智与机器中心）
Many animals, and an increasing number of artificial agents, display sophisticated capabilities to perceive and manipulate objects. But human beings remain distinctive in their capacity for flexible, creative tool use -- using objects in new ways to act on the world, achieve a goal, or solve a problem. To study this type of general physical problem solving, we introduce the Virtual Tools game. In this game, people solve a large range of challenging physical puzzles in just a handful of attempts. We propose that the flexibility of human physical problem solving rests on an ability to imagine th…
Decoupled Physical Modeling and Execution for Physics Reasoning
University of Pennsylvania（宾夕法尼亚大学）；William & Mary（威廉玛丽学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon（亚马逊公司）；Stevens Institute of Technology（史蒂文斯理工学院）；Northwestern University（西北大学）
Objects are a centerpiece of the mathematical realm and our interaction with and reasoning about it, just as they are of the physical one (if not more). And humans' mathematical reasoning must ultimately be grounded in our general intelligence. Yet in contemporary cognitive science and A.I., the physical and mathematical domains are customarily explored separately, which allows for baking in assumptions for what objects are for the system - and missing potential connections. In this paper, I put the issue into its philosophical and cognitive context. I then describe an abstract theoretical fra…
Neural networks lack the ability to reason about qualitative physics and so cannot generalize to scenarios and tasks unseen during training. We propose ESPRIT, a framework for commonsense reasoning about qualitative physics in natural language that generates interpretable descriptions of physical events. We use a two-step approach of first identifying the pivotal physical events in an environment and then generating natural language descriptions of those events using a data-to-text approach. Our framework learns to generate explanations of how the physical simulation will causally evolve so th…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；HKU（香港大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；MIT BCS（麻省理工学院脑与认知科学系）；CBMM（麻省理工学院大脑、心智与机器研究中心）
In this work, we propose a unified framework, called Visual Reasoning with Differ-entiable Physics (VRDP), that can jointly learn visual concepts and infer physics models of objects and their interactions from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses each video frame into object-centric trajectories and represents them as latent scene representations. The concept learner grounds visual concepts (e.g., color, shape, and material) from t…
