Manipulation tasks often require a robot to adjust its sensorimotor skills based on the state it finds itself in. Taking peg-in-hole as an example: once the peg is aligned with the hole, the robot should push the peg downwards. While high level execution frameworks such as state machines and behavior trees are commonly used to formalize such decision-making problems, these frameworks require a mechanism to detect the high-level symbolic state. Handcrafting heuristics to identify symbolic states can be brittle, and using data-driven methods can produce noisy predictions, particularly when working with limited datasets, as is common in real-world robotic scenarios. This paper proposes a Bayesian state estimation method to predict symbolic states with predicate classifiers. This method requires little training data and allows fusing noisy observations from multiple sensor modalities. We evaluate our framework on a set of real-world peg-in-hole and connector-socket insertion tasks, demonstrating its ability to classify symbolic states and to generalize to unseen tasks, outperforming baseline methods. We also demonstrate the ability of our method to improve the robustness of manipulation policies on a real robot.
Symbols representing abstract states such as "dish in dishwasher" or "cup on table" allow robots to reason over long horizons by hiding details unnecessary for high-level planning. Current methods for learning to identify symbolic states in visual data require large amounts of labeled training data, but manually annotating such datasets is prohibitively expensive due to the combinatorial number of predicates in images. We propose a novel method for automatically labeling symbolic states in large-scale video activity datasets by exploiting known pre- and post-conditions of actions. This automat…
Integrated Task and Motion Planning (TMP) provides a promising class of approaches for solving robot planning problems with intricate symbolic and geometric constraints. However, the practical usefulness of TMP planners is limited by their need for symbolic abstractions of robot actions, which are difficult to construct even for experts. We propose an approach to automatically construct and continuously improve a symbolic abstraction of a robot action via observations of the robot performing the action. This approach, called automatic abstraction repair, allows symbolic abstractions to be init…
Unifying Deep Predicate Invention with Pre-trained Foundation Models
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Computer Science and Engineering Division, University of Michigan（密歇根大学计算机科学与工程系）；Department of Computer Science, University of Pittsburgh（匹兹堡大学计算机科学系）；Centaur AI Institute（Centaur人工智能研究所）；Department of Electrical and Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程系）
VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
University of Cambridge（剑桥大学）；Cornell University（康奈尔大学）；Massachusetts Institute of Technology（麻省理工学院）；Princeton University（普林斯顿大学）；University of Oxford（牛津大学）；The Alan Turing Institute（阿兰·图灵研究所）
Lang2Manip: A Tool for LLM-Based Symbolic-to-Geometric Planning for Manipulation
Khalifa University Center for Autonomous Robotic Systems (KUCARS)（卡利法大学自主机器人系统中心）；Khalifa University（卡利法大学）；Institute of Industrial and Control Engineering (IOC)（工业与控制工程研究所）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
Indian Institute of Technology Delhi（德里印度理工学院）；Samsung R&D Institute India, Delhi（三星印度德里研发中心）；Imperial College London（帝国理工学院）
Our goal is to enable a robot to learn how to sequence its actions to perform tasks specified as natural language instructions, given successful demonstrations from a human partner. The ability to plan high-level tasks can be factored as (i) inferring specific goal predicates that characterize the task implied by a language instruction for a given world state and (ii) synthesizing a feasible goal-reaching action-sequence with such predicates. For the former, we leverage a neural network prediction model, while utilizing a symbolic planner for the latter. We introduce a novel neuro-symbolic mod…
The Price Is Not Right: Neuro-Symbolic Methods Outperform VLAs on Structured Long-Horizon Manipulation Tasks with Significantly Lower Energy Consumption
Human-Robot Interaction Lab, Tufts University（Tufts大学人机交互实验室）；AIT Austrian Institute of Technology GmbH（奥地利技术研究所）
Laboratory for Intelligent Decision and Autonomous Robots, George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology（佐治亚理工学院乔治·W·伍德拉夫机械工程学院智能决策与自主机器人实验室）；Georgia Tech Research Institute（佐治亚理工学院研究学院（Georgia Tech Research Institute））
This study proposes a Task and Motion Planning (TAMP) method with symbolic decisions embedded in a bilevel optimization. This TAMP method exploits the discrete structure of sequential manipulation for long-horizon and versatile tasks in dynamically changing environments. At the symbolic planning level, we propose a scalable decision-making method for long-horizon manipulation tasks using the Planning Domain Definition Language (PDDL) with causal graph decomposition. At the motion planning level, we devise a trajectory optimization (TO) approach based on the Alternating Direction Method of Mult…
NVIDIA（英伟达）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
We present a visually grounded hierarchical planning algorithm for long-horizon manipulation tasks. Our algorithm offers a joint framework of neuro-symbolic task planning and low-level motion generation conditioned on the specified goal. At the core of our approach is a two-level scene graph representation, namely geometric scene graph and symbolic scene graph. This hierarchical representation serves as a structured, object-centric abstraction of manipulation scenes. Our model uses graph neural networks to process these scene graphs for predicting high-level task plans and low-level motions. W…
Technical University of Munich（慕尼黑工业大学）；University of Innsbruck（因斯布鲁克大学）；German Aerospace Center (DLR)（德国航空航天中心（DLR））
The definition of symbolic descriptions that consistently represent relevant geometrical aspects in manipulation tasks is a challenging problem that has received little attention in the robotic community. This definition is usually done from an observer perspective of a finite set of object relations and orientations that only satisfy geometrical constraints to execute experiments in laboratory conditions. This restricts the possible changes with manipulation actions in the object configuration space to those compatible with that particular external reference definitions, which greatly limits …
University of Freiburg（弗莱堡大学）；University of Bremen（不来梅大学）
Service robots in the future need to execute abstract instructions such as "fetch the milk from the fridge". To translate such instructions into actionable plans, robots require in-depth background knowledge. With regards to interactions with doors and drawers, robots require articulation models that they can use for state estimation and motion planning. Existing frameworks model articulated connections as abstract concepts such as prismatic, or revolute, but do not provide a parameterized model of these connections for computation. In this paper, we introduce a novel framework that uses symbo…
Neuro-Symbolic Manipulation Understanding with Enriched Semantic Event Chains
Department of Computer Science, School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran（德黑兰大学数学、统计学与计算机科学学院计算机科学系）；Computer Science, College of Science, University of Tehran（德黑兰大学科学学院计算机科学系）
Mobile manipulation problems involving many objects are challenging to solve due to the high dimensionality and multi-modality of their hybrid configuration spaces. Planners that perform a purely geometric search are prohibitively slow for solving these problems because they are unable to factor the configuration space. Symbolic task planners can efficiently construct plans involving many variables but cannot represent the geometric and kinematic constraints required in manipulation. We present the FFRob algorithm for solving task and motion planning problems. First, we introduce Extended Acti…
Synchronizing expectations and knowledge about the state of the world is an essential capability for effective collaboration. For robots to effectively collaborate with humans and other autonomous agents, it is critical that they be able to generate intelligible explanations to reconcile differences between their understanding of the world and that of their collaborators. In this work we present Single-shot Policy Explanation for Augmenting Rewards (SPEAR), a novel sequential optimization algorithm that uses semantic explanations derived from combinations of planning predicates to augment agen…
Mobile robots, performing long-term manipulation activities in human environments, have to perceive a wide variety of objects possessing very different visual characteristics and need to reliably keep track of these throughout the execution of a task. In order to be efficient, robot perception capabilities need to go beyond what is currently perceivable and should be able to answer queries about both current and past scenes. In this paper we investigate a perception system for long-term robot manipulation that keeps track of the changing environment and builds a representation of the perceived…
While robots can learn models to solve many manipulation tasks from raw visual input, they cannot usually use these models to solve new problems. On the other hand, symbolic planning methods such as STRIPS have long been able to solve new problems given only a domain definition and a symbolic goal, but these approaches often struggle on the real world robotic tasks due to the challenges of grounding these symbols from sensor data in a partially-observable world. We propose Deep Planning Domain Learning (DPDL), an approach that combines the strengths of both methods to learn a hierarchical mode…
