We present Points2Plans, a framework for composable planning with a relational dynamics model that enables robots to solve long-horizon manipulation tasks from partial-view point clouds. Given a language instruction and a point cloud of the scene, our framework initiates a hierarchical planning procedure, whereby a language model generates a high-level plan and a sampling-based planner produces constraint-satisfying continuous parameters for manipulation primitives sequenced according to the high-level plan. Key to our approach is the use of a relational dynamics model as a unifying interface between the continuous and symbolic representations of states and actions, thus facilitating language-driven planning from high-dimensional perceptual input such as point clouds. Whereas previous relational dynamics models require training on datasets of multi-step manipulation scenarios that align with the intended test scenarios, Points2Plans uses only single-step simulated training data while generalizing zero-shot to a variable number of steps during real-world evaluations. We evaluate our approach on tasks involving geometric reasoning, multi-object interactions, and occluded object reasoning in both simulated and real-world settings. Results demonstrate that Points2Plans offers strong generalization to unseen long-horizon tasks in the real world, where it solves over 85% of evaluated tasks while the next best baseline solves only 50%.
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗 G. 艾伦计算机科学与工程学院）；NVIDIA（英伟达）
Learning high-level navigation behaviors has important implications: it enables robots to build compact visual memory for repeating demonstrations and to build sparse topological maps for planning in novel environments. Existing approaches only learn discrete, short-horizon behaviors. These standalone behaviors usually assume a discrete action space with simple robot dynamics, thus they cannot capture the intricacy and complexity of real-world trajectories. To this end, we propose Composable Behavior Embedding (CBE), a continuous behavior representation for long-horizon visual navigation. CBE …
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
Shanghai AI Laboratory（上海人工智能实验室）；School of Software, Beihang University（北京航空航天大学软件学院）；Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）
We present a hybrid compositional approach for real-time mission planning for multi-rotor unmanned aerial vehicles (UAVs) in a time critical search and rescue scenario. Starting with a known environment, we specify the mission using Metric Temporal Logic (MTL) and use a hybrid dynamical model to capture the various modes of UAV operation. We then divide the mission into several sub-tasks by exploiting the invariant nature of safety and timing constraints along the way, and the different modes (i.e., dynamics) of the UAV. For each sub-task, we translate the MTL specifications into linear constr…
Objects rarely sit in isolation in human environments. As such, we'd like our robots to reason about how multiple objects relate to one another and how those relations may change as the robot interacts with the world. To this end, we propose a novel graph neural network framework for multi-object manipulation to predict how inter-object relations change given robot actions. Our model operates on partial-view point clouds and can reason about multiple objects dynamically interacting during the manipulation. By learning a dynamics model in a learned latent graph embedding space, our model enable…
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
In robotic domains, learning and planning are complicated by continuous state spaces, continuous action spaces, and long task horizons. In this work, we address these challenges with Neuro-Symbolic Relational Transition Models (NSRTs), a novel class of models that are data-efficient to learn, compatible with powerful robotic planning methods, and generalizable over objects. NSRTs have both symbolic and neural components, enabling a bilevel planning scheme where symbolic AI planning in an outer loop guides continuous planning with neural models in an inner loop. Experiments in four robotic plan…
Rey Juan Carlos University（胡安·卡洛斯国王大学）；CEA list（法国原子能委员会列表研究所）；University of Leon（莱昂大学）
Robots need task planning to sequence and execute actions toward achieving their goals. On the other hand, Behavior Trees provide a mathematical model for specifying plan execution in an intrinsically composable, reactive, and robust way. PDDL (Planning Domain Definition Language) has become the standard description language for most planners. In this paper, we present a novel algorithm to systematically create behavior trees from PDDL plans to execute them. This approach uses the execution graph of the plan to generate a behavior tree. The most remarkable contribution of this approach is the …
The Chinese University of Hong Kong（香港中文大学）；Xi’an Jiaotong University（西安交通大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）
Embodied AI is an inevitable trend that emphasizes the interaction between intelligent entities and the real world, with broad applications in Robotics, especially target-driven navigation. This task requires the robot to find an object of a certain category efficiently in an unknown domestic environment. Recent works focus on exploiting layout relationships by graph neural networks (GNNs). However, most of them obtain robot actions directly from observations in an end-to-end manner via an incomplete relation graph, which is not interpretable and reliable. We decouple this task and propose ReV…
Honda Research Institute (HRI) Europe（本田欧洲研究院（HRI Europe））；Technical University of Darmstadt（达姆施塔特工业大学）
This work addresses the task of risk evaluation in traffic scenarios with limited observability due to restricted sensorial coverage. Here, we concentrate on intersection scenarios that are difficult to access visually. To identify the area of sight, we employ ray casting on a local dynamic map providing geometrical information and road infrastructure. Based on the area with reduced visibility, we first model scene entities that pose a potential risk without being visually perceivable yet. Then, we predict a worst-case trajectory in the survival analysis for collision risk estimation. Resultin…
Honda Research Institute, USA（本田美国研究所）；University of California, Berkeley（加州大学伯克利分校）
Multi-agent interacting systems are prevalent in the world, from pure physical systems to complicated social dynamic systems. In many applications, effective understanding of the situation and accurate trajectory prediction of interactive agents play a significant role in downstream tasks, such as decision making and planning. In this paper, we propose a generic trajectory forecasting framework (named EvolveGraph) with explicit relational structure recognition and prediction via latent interaction graphs among multiple heterogeneous, interactive agents. Considering the uncertainty of future be…
Multi-Agent Dynamic Relational Reasoning for Social Robot Navigation
University of California, Riverside（加州大学河滨分校）；University of California, Berkeley（加州大学伯克利分校）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Stanford University（斯坦福大学）
