The ability to perceive and reason about social interactions in the context of physical environments is core to human social intelligence and human-machine cooperation. However, no prior dataset or benchmark has systematically evaluated physically grounded perception of complex social interactions that go beyond short actions, such as high-fiving, or simple group activities, such as gathering. In this work, we create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions by including social concepts such as helping another agent. PHASE consists of 2D animations of pairs of agents moving in a continuous space generated procedurally using a physics engine and a hierarchical planner. Agents have a limited field of view, and can interact with multiple objects, in an environment that has multiple landmarks and obstacles. Using PHASE, we design a social recognition task and a social prediction task. PHASE is validated with human experiments demonstrating that humans perceive rich interactions in the social events, and that the simulated agents behave similarly to humans. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE (SIMulation, Planning and Local Estimation), which outperforms state-of-the-art feed-forward neural networks. We hope that PHASE can serve as a difficult new challenge for developing new models that can recognize complex social interactions.
Learning physically grounded traffic accident reconstruction from public accident reports
State Key Laboratory of Internet of Things for Smart City（物联网智能城市国家重点实验室）；Department of Civil Engineering（土木工程系）；Department of Computer and Information Science（计算机与信息科学系）
Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields
Institute for AI, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理学与认知科学学院）；School of EECS, Peking University（北京大学电子工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Yuanpei College, Peking University（北京大学元培学院）；State Key Lab of General AI, Peking University（北京大学通用人工智能国家重点实验室）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京行为与心理健康重点实验室）
ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion
Mechatronics, Systems and Control Lab (MSC Lab), Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院机械工程系机电一体化、系统与控制实验室（MSC实验室））；Samsung Electronics, Future Robotics AI Group（三星电子未来机器人人工智能集团）；School of Electrical Engineering, Hanyang University（汉阳大学电气工程学院）；Advanced Robotics Research Center, Korea Institute of Machinery & Materials (KIMM)（韩国机械与材料研究所先进机器人研究中心）
It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it me…
There exist well-developed frameworks for causal modelling, but these require rather a lot of human domain expertise to define causal variables and perform interventions. In order to enable autonomous agents to learn abstract causal models through interactive experience, the existing theoretical foundations need to be extended and clarified. Existing frameworks give no guidance regarding variable choice / representation, and more importantly, give no indication as to which behaviour policies or physical transformations of state space shall count as interventions. The framework sketched in this…
ChronoAgentic: A Code-based Multi-Agent World Simulator for Physically Grounded Simulation Construction
Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison（威斯康星大学麦迪逊分校机械与航空航天工程系）；School of Computer, Data, and Information Sciences, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机、数据与信息科学学院）
Although virtual agents are increasingly situated in environments where natural language is the most effective mode of interaction with humans, these exchanges are rarely used as an opportunity for learning. Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding: semantic parsers built on top of fixed object categories are precise but inflexible and end-to-end models are maximally expressive, but fickle and opaque. Our goal is to develop a system that balances the strengths of each approach so that users can teach ag…
DeepMind（深度思维）；National University of Singapore（新加坡国立大学）
Recent work in deep reinforcement learning (RL) has produced algorithms capable of mastering challenging games such as Go, chess, or shogi. In these works the RL agent directly observes the natural state of the game and controls that state directly with its actions. However, when humans play such games, they do not just reason about the moves but also interact with their physical environment. They understand the state of the game by looking at the physical board in front of them and modify it by manipulating pieces using touch and fine-grained motor control. Mastering complicated physical syst…
Intelligent robots need to achieve abstract objectives using concrete, spatiotemporally complex sensory information and motor control. Tabula rasa deep reinforcement learning (RL) has tackled demanding tasks in terms of either visual, abstract, or physical reasoning, but solving these jointly remains a formidable challenge. One recent, unsolved benchmark task that integrates these challenges is Mujoban, where a robot needs to arrange 3D warehouses generated from 2D Sokoban puzzles. We explore whether integrated tasks like Mujoban can be solved by composing RL modules together in a sense-plan-a…
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
Given a simple request like Put a washed apple in the kitchen fridge, humans can reason in purely abstract terms by imagining action sequences and scoring their likelihood of success, prototypicality, and efficiency, all without moving a muscle. Once we see the kitchen in question, we can update our abstract plans to fit the scene. Embodied agents require the same abilities, but existing work does not yet provide the infrastructure necessary for both reasoning abstractly and executing concretely. We address this limitation by introducing ALFWorld, a simulator that enables agents to learn abstr…
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；McMaster University（麦克马斯特大学）；Meta Platforms（Meta平台公司）
By capturing statistical patterns in large corpora, machine learning has enabled significant advances in natural language processing, including in machine translation, question answering, and sentiment analysis. However, for agents to intelligently interact with humans, simply capturing the statistical patterns is insufficient. In this paper we investigate if, and how, grounded compositional language can emerge as a means to achieve goals in multi-agent populations. Towards this end, we propose a multi-agent learning environment and learning methods that bring about emergence of a basic compos…
