Robotic manipulation involves actions where contacts occur between the robot and the objects. In this scope, the availability of physics-based engines allows motion planners to comprise dynamics between rigid bodies, which is necessary for planning this type of actions. However, physics-based motion planning is computationally intensive due to the high dimensionality of the state space and the need to work with a low integration step to find accurate solutions. On the other hand, manipulation actions change the environment and conditions further actions and motions. To cope with this issue, the representation of manipulation actions using ontologies enables a semantic-based inference process that alleviates the computational cost of motion planning. This paper proposes a manipulation planning framework where physics-based motion planning is enhanced with ontological knowledge representation and reasoning. The proposal has been implemented and is illustrated and validated with a simple example. Its use in grasping tasks in cluttered environments is currently under development.
This work introduces PokeRRT, a novel motion planning algorithm that demonstrates poking as an effective non-prehensile manipulation skill to enable fast manipulation of objects and increase the size of a robot's reachable workspace. Our qualitative and quantitative results demonstrate the advantages of poking over pushing and grasping in planning object trajectories through uncluttered and cluttered environments.
The path planning problems arising in manipulation planning and in task and motion planning settings are typically repetitive: the same manipulator moves in a space that only changes slightly. Despite this potential for reuse of information, few planners fully exploit the available information. To better enable this reuse, we decompose the collision checking into reusable, and non-reusable parts. We then treat the sequences of path planning problems in manipulation planning as a multiquery path planning problem. This allows the usage of planners that actively minimize planning effort over mult…
Manipulation in clutter requires solving complex sequential decision making problems in an environment rich with physical interactions. The transfer of motion planning solutions from simulation to the real world, in open-loop, suffers from the inherent uncertainty in modelling real world physics. We propose interleaving planning and execution in real-time, in a closed-loop setting, using a Receding Horizon Planner (RHP) for pushing manipulation in clutter. In this context, we address the problem of finding a suitable value function based heuristic for efficient planning, and for estimating the…
We present a damage-aware planning approach which determines the best sequence to manipulate a number of objects in a scene. This works on task-planning level, abstracts from motion planning and anticipates the dynamics of the scene using a physics simulation. Instead of avoiding interaction with the environment, we take unintended motion of other objects into account and plan manipulation sequences which minimize the potential damage. Our method can also be used as a validation measure to judge planned motions for their feasibility in terms of damage avoidance. We evaluate our approach on one…
In robot manipulation, planning the motion of a robot manipulator to grasp an object is a fundamental problem. A manipulation planner needs to generate a trajectory of the manipulator arm to avoid obstacles in the environment and plan an end-effector pose for grasping. While trajectory planning and grasp planning are often tackled separately, how to efficiently integrate the two planning problems remains a challenge. In this work, we present a novel method for joint motion and grasp planning. Our method integrates manipulation trajectory optimization with online grasp synthesis and selection, …
Lang2Manip: A Tool for LLM-Based Symbolic-to-Geometric Planning for Manipulation
Khalifa University Center for Autonomous Robotic Systems (KUCARS)（卡利法大学自主机器人系统中心）；Khalifa University（卡利法大学）；Institute of Industrial and Control Engineering (IOC)（工业与控制工程研究所）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
University Stuttgart（斯图加特大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Logic-Geometric Programming (LGP) is a powerful motion and manipulation planning framework, which represents hierarchical structure using logic rules that describe discrete aspects of problems, e.g., touch, grasp, hit, or push, and solves the resulting smooth trajectory optimization. The expressive power of logic allows LGP for handling complex, large-scale sequential manipulation and tool-use planning problems. In this paper, we extend the LGP formulation to stochastic domains. Based on the control-inference duality, we interpret LGP in a stochastic domain as fitting a mixture of Gaussians to…
Dexterous manipulation tasks often require contact switching, where fingers make and break contact with the object. We propose a method that plans trajectories for dexterous manipulation tasks involving contact switching using contact-implicit trajectory optimization (CITO) augmented with a high-level discrete contact sequence planner. We first use the high-level planner to find a sequence of finger contact switches given a desired object trajectory. With this contact sequence plan, we impose additional constraints in the CITO problem. We show that our method finds trajectories approximately 7…
KTH Royal Institute of Technology（瑞典皇家理工学院）；Yale University（耶鲁大学）
This paper addresses non-prehensile rearrangement planning problems where a robot is tasked to rearrange objects among obstacles on a planar surface. We present an efficient planning algorithm that is designed to impose few assumptions on the robot's non-prehensile manipulation abilities and is simple to adapt to different robot embodiments. For this, we combine sampling-based motion planning with reinforcement learning and generative modeling. Our algorithm explores the composite configuration space of objects and robot as a search over robot actions, forward simulated in a physics model. Thi…
The empirical success of Reinforcement Learning (RL) in the setting of contact-rich manipulation leaves much to be understood from a model-based perspective, where the key difficulties are often attributed to (i) the explosion of contact modes, (ii) stiff, non-smooth contact dynamics and the resulting exploding / discontinuous gradients, and (iii) the non-convexity of the planning problem. The stochastic nature of RL addresses (i) and (ii) by effectively sampling and averaging the contact modes. On the other hand, model-based methods have tackled the same challenges by smoothing contact dynami…
Faculty of Electrical and Computer Engineering, University of Zagreb（萨格勒布大学电气与计算机工程学院）
This paper presents a method for aerial manipulator end-effector trajectory tracking by encompassing dynamics of the Unmanned Aerial Vehicle (UAV) and null space of the manipulator attached to it in the motion planning procedure. The proposed method runs in phases. Trajectory planning starts by not accounting for roll and pitch angles of the underactuated UAV system. Next, we propose simulating the dynamics on such a trajectory and obtaining UAV attitude through the model. The full aerial manipulator state obtained in such a manner is further utilized to account for discrepancies in planned an…
Laboratory for Intelligent Decision and Autonomous Robots, George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology（佐治亚理工学院乔治·W·伍德拉夫机械工程学院智能决策与自主机器人实验室）；Georgia Tech Research Institute（佐治亚理工学院研究学院（Georgia Tech Research Institute））
This study proposes a Task and Motion Planning (TAMP) method with symbolic decisions embedded in a bilevel optimization. This TAMP method exploits the discrete structure of sequential manipulation for long-horizon and versatile tasks in dynamically changing environments. At the symbolic planning level, we propose a scalable decision-making method for long-horizon manipulation tasks using the Planning Domain Definition Language (PDDL) with causal graph decomposition. At the motion planning level, we devise a trajectory optimization (TO) approach based on the Alternating Direction Method of Mult…
We present an optimization-based motion planning algorithm to compute a smooth, collision-free trajectory for a manipulator used to transfer a liquid from a source to a target container. We take into account fluid dynamics constraints as part of trajectory computation. In order to avoid the high complexity of exact fluid simulation, we introduce a simplified dynamics model based on physically inspired approximations and system identification. Our optimization approach can incorporate various other constraints such as collision avoidance with the obstacles, kinematic and dynamics constraints of…
Mobile manipulation problems involving many objects are challenging to solve due to the high dimensionality and multi-modality of their hybrid configuration spaces. Planners that perform a purely geometric search are prohibitively slow for solving these problems because they are unable to factor the configuration space. Symbolic task planners can efficiently construct plans involving many variables but cannot represent the geometric and kinematic constraints required in manipulation. We present the FFRob algorithm for solving task and motion planning problems. First, we introduce Extended Acti…
Combining Sampling- and Gradient-based Planning for Contact-rich Manipulation
Politecnico di Milano（米兰理工大学）；Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA)（达勒·莫勒人工智能研究所）；Scuola Universitaria Professionale della Svizzera Italiana (SUPSI)（瑞士意大利语区应用科学大学）；Università della Svizzera Italiana (USI)（瑞士意大利语区大学）；Fraunhofer IPK（弗劳恩霍夫制造工程和自动化研究所）
We focus on the task of object manipulation to an arbitrary goal pose, in which a robot is supposed to pick an assigned object to place at the goal position with a specific orientation. However, limited by the execution space of the manipulator with gripper, one-step picking, moving and releasing might be failed, where a reorientation object pose is required as a transition. In this paper, we propose a learning-driven anytime prioritized search-based solver to find a feasible solution with low path cost in a short time. In our work, the problem is formulated as a hierarchical learning problem,…
The State University of New York at Binghamton（纽约州立大学宾汉姆顿分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能研究院）
Task and motion planning (TAMP) algorithms aim to help robots achieve task-level goals, while maintaining motion-level feasibility. This paper focuses on TAMP domains that involve robot behaviors that take extended periods of time (e.g., long-distance navigation). In this paper, we develop a visual grounding approach to help robots probabilistically evaluate action feasibility, and introduce a TAMP algorithm, called GROP, that optimizes both feasibility and efficiency. We have collected a dataset that includes 96,000 simulated trials of a robot conducting mobile manipulation tasks, and then us…
Hierarchical Deformation Planning and Neural Tracking for DLOs in Constrained Environments
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong（机械与自动化工程系，香港中文大学）；Multi-scale Medical Robotics Centre（多尺度医学机器人中心）
Edinburgh Centre for Robotics（爱丁堡机器人中心）；University of Edinburgh（爱丁堡大学）；Heriot-Watt University（赫瑞瓦特大学）；Rice University（莱斯大学）
Robotic systems may frequently come across similar manipulation planning problems that result in similar motion plans. Instead of planning each problem from scratch, it is preferable to leverage previously computed motion plans, i.e., experiences, to ease the planning. Different approaches have been proposed to exploit prior information on novel task instances. These methods, however, rely on a vast repertoire of experiences and fail when none relates closely to the current problem. Thus, an open challenge is the ability to generalise prior experiences to task instances that do not necessarily…
LAGO Policy: Latency-Aware Asynchronous Diffusion Policies with Goal-Directed Collision-Free Planning for Smooth Manipulation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；International Digital Economy Academy（国际数字经济学院）；The University of Hong Kong（香港大学）；Southern University of Science and Technology（南方科技大学）
Whole-body motion planning and safety-critical control for aerial manipulation
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机械与航空航天工程学院，南洋理工大学，新加坡）；Department of Aerospace Engineering, Seoul National University, South Korea（航空航天工程系，首尔国立大学，韩国）
Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院）；Miraca Research Institute G. K.（Miraca研究所株式会社）
The paper develops a robotic manipulation system to treat the pressing needs for handling a large number of test tubes in clinical examination and replace or reduce human labor. It presents the technical details of the system, which separates and arranges test tubes in racks with the help of 3D vision and artificial intelligence (AI) reasoning/planning. The developed system only requires a person to put a rack with mixed and non-arranged tubes in front of a robot. The robot autonomously performs recognition, reasoning, planning, manipulation, etc., and returns a rack with separated and arrange…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Toyota Technological Institute（丰田技术研究院）
This report describes our winning submission to the Real Robot Challenge (https://real-robot-challenge.com/). The Real Robot Challenge is a three-phase dexterous manipulation competition that involves manipulating various rectangular objects with the TriFinger Platform. Our approach combines motion planning with several motion primitives to manipulate the object. For Phases 1 and 2, we additionally learn a residual policy in simulation that applies corrective actions on top of our controller. Our approach won first place in Phase 2 and Phase 3 of the competition. We were anonymously known as `…
Applied AI Lab (A2I)（应用人工智能实验室）；Dynamic Robot Systems (DRS)（动态机器人系统实验室）；Oxford Robotics Institute (ORI), University of Oxford（牛津大学牛津机器人研究所）
We present a novel approach to path planning for robotic manipulators, in which paths are produced via iterative optimisation in the latent space of a generative model of robot poses. Constraints are incorporated through the use of constraint satisfaction classifiers operating on the same space. Optimisation leverages gradients through our learned models that provide a simple way to combine goal reaching objectives with constraint satisfaction, even in the presence of otherwise non-differentiable constraints. Our models are trained in a task-agnostic manner on randomly sampled robot poses. In …
We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable obj…
Massachusetts Institute of Technology（麻省理工学院）；KAIST Graduate School of AI（韩国科学技术院人工智能研究生院）；Samsung AI Center Montreal（三星蒙特利尔人工智能中心）
We present a framework for solving long-horizon planning problems involving manipulation of rigid objects that operates directly from a point-cloud observation, i.e. without prior object models. Our method plans in the space of object subgoals and frees the planner from reasoning about robot-object interaction dynamics by relying on a set of generalizable manipulation primitives. We show that for rigid bodies, this abstraction can be realized using low-level manipulation skills that maintain sticking contact with the object and represent subgoals as 3D transformations. To enable generalization…
Real-world manipulation problems in heavy clutter require robots to reason about potential contacts with objects in the environment. We focus on pick-and-place style tasks to retrieve a target object from a shelf where some `movable' objects must be rearranged in order to solve the task. In particular, our motivation is to allow the robot to reason over and consider non-prehensile rearrangement actions that lead to complex robot-object and object-object interactions where multiple objects might be moved by the robot simultaneously, and objects might tilt, lean on each other, or topple. To supp…
Trajectory Planning and Control for Robotic Magnetic Manipulation
Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University（电气工程与自动化系，电气工程学院，阿alto大学）；Faculty of Engineering and Natural Sciences, Automation Technology and Mechanical Engineering, Tampere University（工程与自然科学学院，自动化技术与机械工程，塔尔基大学）
Robot manipulation in cluttered environments often requires complex and sequential rearrangement of multiple objects in order to achieve the desired reconfiguration of the target objects. Due to the sophisticated physical interactions involved in such scenarios, rearrangement-based manipulation is still limited to a small range of tasks and is especially vulnerable to physical uncertainties and perception noise. This paper presents a planning framework that leverages the efficiency of sampling-based planning approaches, and closes the manipulation loop by dynamically controlling the planning h…
This paper presents Contact Mode Guided Manipulation Planning (CMGMP) for 3D quasistatic and quasidynamic rigid body motion planning in dexterous manipulation. The CMGMP algorithm generates hybrid motion plans including both continuous state transitions and discrete contact mode switches, without the need for pre-specified contact sequences or pre-designed motion primitives. The key idea is to use automatically enumerated contact modes of environment-object contacts to guide the tree expansions during the search. Contact modes automatically synthesize manipulation primitives, while the samplin…
CNRS（法国国家科学研究中心）；LAAS（系统分析与体系结构实验室）；Courant Institute, NYU（纽约大学库朗研究所）
Consider the problem of planning collision-free motion of $n$ objects in the plane movable through contact with a robot that can autonomously translate in the plane and that can move a maximum of $m \leq n$ objects simultaneously. This represents the abstract formulation of a manipulation planning problem that is proven to be decidable in this paper. The tools used for proving decidability of this simplified manipulation planning problem are, in fact, general enough to handle the decidability problem for the wider class of systems characterized by a stratified configuration space. These includ…
Tandon School of Engineering, New York University（纽约大学坦登工程学院）
This paper presents an efficient approach to object manipulation planning using Monte Carlo Tree Search (MCTS) to find contact sequences and an efficient ADMM-based trajectory optimization algorithm to evaluate the dynamic feasibility of candidate contact sequences. To accelerate MCTS, we propose a methodology to learn a goal-conditioned policy-value network to direct the search towards promising nodes. Further, manipulation-specific heuristics enable to drastically reduce the search space. Systematic object manipulation experiments in a physics simulator and on real hardware demonstrate the e…
The discontinuities and multi-modality introduced by contacts make manipulation planning challenging. Many previous works avoid this problem by pre-designing a set of high-level motion primitives like grasping and pushing. However, such motion primitives are often not adequate to describe dexterous manipulation motions. In this work, we propose a method for dexterous manipulation planning at a more primitive level. The key idea is to use contact modes to guide the search in a sampling-based planning framework. Our method can automatically generate contact transitions and motion trajectories un…
Humanoid Loco-manipulation Planning based on Graph Search and Reachability Maps
CNRS-AIST JRL (Joint Robotics Laboratory)（CNRS-AIST联合机器人实验室）；IRL（机器人研究所）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）
Osaka University（大阪大学）；National Inst. of AIST（国立产业技术综合研究所）
This paper presents a manipulation planning algorithm for robots to reorient objects. It automatically finds a sequence of robot motion that manipulates and prepares an object for specific tasks. Examples of the preparatory manipulation planning problems include reorienting an electric drill to cut holes, reorienting workpieces for assembly, and reorienting cargo for packing, etc. The proposed algorithm could plan single and dual arm manipulation sequences to solve the problems. The mechanism under the planner is a regrasp graph which encodes grasp configurations and object poses. The algorith…
