Use of physics-based simulation as a planning model enables a planner to reason and generate plans that involve non-trivial interactions with the world. For example, grasping a milk container out of a cluttered refrigerator may involve moving a robot manipulator in between other objects, pushing away the ones that are movable and avoiding interactions with certain fragile containers. A physics-based simulator allows a planner to reason about the effects of interactions with these objects and to generate a plan that grasps the milk container successfully. The use of physics-based simulation for planning however is underutilized. One of the reasons for it being that physics-based simulations are typically way too slow for being used within a planning loop that typically requires tens of thousands of actions to be evaluated within a matter of a second or two. In this work, we develop a planning algorithm that tries to address this challenge. In particular, it builds on the observation that only a small number of actions actually need to be simulated using physics, and the remaining set of actions, such as moving an arm around obstacles, can be evaluated using a much simpler internal planning model, e.g., a simple collision-checking model. Motivated by this, we develop an algorithm called Planning with Selective Physics-based Simulation that automatically discovers what should be simulated with physics and what can utilize an internal planning model for pick-and-place tasks.
Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）
Recently, haptic gloves have been extensively explored for various practical applications, such as manipulation learning. Previous glove devices have different force-driven systems, such as shape memory alloys, servo motors and pneumatic actuators; however, these proposed devices may have difficulty in fast finger movement, easy reproduction, and safety issues. In this study, we propose MagGlove, a novel haptic glove with a movable magnet mechanism that has a linear motor, to solve these issues. The proposed MagGlove device is a compact system on the back of the wearer's hand with high respons…
Real-world manipulation problems in heavy clutter require robots to reason about potential contacts with objects in the environment. We focus on pick-and-place style tasks to retrieve a target object from a shelf where some `movable' objects must be rearranged in order to solve the task. In particular, our motivation is to allow the robot to reason over and consider non-prehensile rearrangement actions that lead to complex robot-object and object-object interactions where multiple objects might be moved by the robot simultaneously, and objects might tilt, lean on each other, or topple. To supp…
In manipulation tasks, a robot interacts with movable object(s). The configuration space in manipulation planning is thus the Cartesian product of the configuration space of the robot with those of the movable objects. It is the complex structure of such a "Composite Configuration Space" that makes manipulation planning particularly challenging. Previous works approximate the connectivity of the Composite Configuration Space by means of discretization or by creating random roadmaps. Such approaches involve an extensive pre-processing phase, which furthermore has to be re-done each time the env…
Robot manipulation in cluttered scenes often requires contact-rich interactions with objects. It can be more economical to interact via non-prehensile actions, for example, push through other objects to get to the desired grasp pose, instead of deliberate prehensile rearrangement of the scene. For each object in a scene, depending on its properties, the robot may or may not be allowed to make contact with, tilt, or topple it. To ensure that these constraints are satisfied during non-prehensile interactions, a planner can query a physics-based simulator to evaluate the complex multi-body intera…
In this paper, we examine the problem of visibility-aware robot navigation among movable obstacles (VANAMO). A variant of the well-known NAMO robotic planning problem, VANAMO puts additional visibility constraints on robot motion and object movability. This new problem formulation lifts the restrictive assumption that the map is fully visible and the object positions are fully known. We provide a formal definition of the VANAMO problem and propose the Look and Manipulate Backchaining (LaMB) algorithm for solving such problems. LaMB has a simple vision-based API that makes it more easily transf…
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
We are interested in pick-and-place style robot manipulation tasks in cluttered and confined 3D workspaces among movable objects that may be rearranged by the robot and may slide, tilt, lean or topple. A recently proposed algorithm, M4M, determines which objects need to be moved and where by solving a Multi-Agent Pathfinding MAPF abstraction of this problem. It then utilises a nonprehensile push planner to compute actions for how the robot might realise these rearrangements and a rigid body physics simulator to check whether the actions satisfy physics constraints encoded in the problem. Howev…
NVIDIA（英伟达）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
In many applications, including logistics and manufacturing, robot manipulators operate in semi-structured environments alongside humans or other robots. These environments are largely static, but they may contain some movable obstacles that the robot must avoid. Manipulation tasks in these applications are often highly repetitive, but require fast and reliable motion planning capabilities, often under strict time constraints. Existing preprocessing-based approaches are beneficial when the environments are highly-structured, but their performance degrades in the presence of movable obstacles, …
Efficient Navigation Among Movable Obstacles using a Mobile Manipulator via Hierarchical Policy Learning
School of Computing in Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院计算机学院）；Robotics Program at the same institute, KAIST（韩国科学技术院机器人项目）
CNRS（法国国家科学研究中心）；LAAS（系统分析与体系结构实验室）；Courant Institute, NYU（纽约大学库朗研究所）
Consider the problem of planning collision-free motion of $n$ objects in the plane movable through contact with a robot that can autonomously translate in the plane and that can move a maximum of $m \leq n$ objects simultaneously. This represents the abstract formulation of a manipulation planning problem that is proven to be decidable in this paper. The tools used for proving decidability of this simplified manipulation planning problem are, in fact, general enough to handle the decidability problem for the wider class of systems characterized by a stratified configuration space. These includ…
Mobile manipulation problems involving many objects are challenging to solve due to the high dimensionality and multi-modality of their hybrid configuration spaces. Planners that perform a purely geometric search are prohibitively slow for solving these problems because they are unable to factor the configuration space. Symbolic task planners can efficiently construct plans involving many variables but cannot represent the geometric and kinematic constraints required in manipulation. We present the FFRob algorithm for solving task and motion planning problems. First, we introduce Extended Acti…
Bench-Push: Benchmarking Pushing-based Navigation and Manipulation Tasks for Mobile Robots
Institute for Robotics and Intelligent Machines, Georgia Institute of Technology（机器人与智能机器研究所，佐治亚理工学院）；Department of Electrical and Computer Engineering, University of Waterloo（电气与计算机工程系，滑铁卢大学）；Department of Mechanical Engineering, University of Alberta（机械工程系，阿尔伯塔大学）
SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学港大剑客基金会数据科学研究所）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电气与电子工程系）
Carnegie Mellon University（卡内基梅隆大学）；National University of Singapore（新加坡国立大学）
This work addresses the challenge of a robot using real-time feedback from contact sensors to reliably manipulate a movable object on a cluttered tabletop. We formulate contact manipulation as a partially observable Markov decision process (POMDP) in the joint space of robot configurations and object poses. The POMDP formulation enables the robot to actively gather information and reduce the uncertainty on the object pose. Further, it incorporates all major constraints for robot manipulation: kinematic reachability, self-collision, and collision with obstacles. To solve the POMDP, we apply DES…
University of Pennsylvania（宾夕法尼亚大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
In this paper we address mobile manipulation planning problems in the presence of sensing and environmental uncertainty. In particular, we consider mobile sensing manipulators operating in environments with unknown geometry and uncertain movable objects, while being responsible for accomplishing tasks requiring grasping and releasing objects in a logical fashion. Existing algorithms either do not scale well or neglect sensing and/or environmental uncertainty. To face these challenges, we propose a hybrid control architecture, where a symbolic controller generates high-level manipulation comman…
Complex manipulation tasks, such as rearrangement planning of numerous objects, are combinatorially hard problems. Existing algorithms either do not scale well or assume a great deal of prior knowledge about the environment, and few offer any rigorous guarantees. In this paper, we propose a novel hybrid control architecture for achieving such tasks with mobile manipulators. On the discrete side, we enrich a temporal logic specification with mobile manipulation primitives such as moving to a point, and grasping or moving an object. Such specifications are translated to an automaton representati…
Non-prehensile manipulation methods usually use a simple end effector, e.g., a single rod, to manipulate the object. Compared to the grasping method, such an end effector is compact and flexible, and hence it can perform tasks in a constrained workspace; As a trade-off, it has relatively few degrees of freedom (DoFs), resulting in an under-actuation problem with complex constraints for planning and control. This paper proposes a new non-prehensile manipulation method for the task of object retrieval in cluttered environments, using a rod-like pusher. Specifically, a candidate trajectory in a c…
STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation
Beijing Institute of Technology（北京理工大学）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology（哈尔滨工业大学）；East China Normal University（华东师范大学）；DeepCybo
In this paper, we present a general approach to automatically visual-servo control the position and shape of a deformable object whose deformation parameters are unknown. The servo-control is achieved by online learning a model mapping between the robotic end-effector's movement and the object's deformation measurement. The model is learned using the Gaussian Process Regression (GPR) to deal with its highly nonlinear property, and once learned, the model is used for predicting the required control at each time step. To overcome GPR's high computational cost while dealing with long manipulation…
This paper presents an omnidirectional aerial manipulation platform for robust and responsive interaction with unstructured environments, toward the goal of contact-based inspection. The fully actuated tilt-rotor aerial system is equipped with a rigidly mounted end-effector, and is able to exert a 6 degree of freedom force and torque, decoupling the system's translational and rotational dynamics, and enabling precise interaction with the environment while maintaining stability. An impedance controller with selective apparent inertia is formulated to permit compliance in certain degrees of free…
University of California, Berkeley（加州大学伯克利分校）；Idiap Research Institute（Idiap研究院）
Generalizing manipulation skills to new situations requires extracting invariant patterns from demonstrations. For example, the robot needs to understand the demonstrations at a higher level while being invariant to the appearance of the objects, geometric aspects of objects such as its position, size, orientation and viewpoint of the observer in the demonstrations. In this paper, we propose an algorithm that learns a joint probability density function of the demonstrations with invariant formulations of hidden semi-Markov models to extract invariant segments (also termed as sub-goals or optio…
We address the problem of visually guided rearrangement planning with many movable objects, i.e., finding a sequence of actions to move a set of objects from an initial arrangement to a desired one, while relying on visual inputs coming from an RGB camera. To do so, we introduce a complete pipeline relying on two key contributions. First, we introduce an efficient and scalable rearrangement planning method, based on a Monte-Carlo Tree Search exploration strategy. We demonstrate that because of its good trade-off between exploration and exploitation our method (i) scales well with the number of…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Deep reinforcement learning trains neural networks using experiences sampled from the replay buffer, which is commonly updated at each time step. In this paper, we propose a method to update the replay buffer adaptively and selectively to train a robot arm to accomplish a suction task in simulation. The response time of the agent is thoroughly taken into account. The state transitions that remain stuck at the boundary of constraint are not stored. The policy trained with our method works better than the one with the common replay buffer update method. The result is demonstrated both by simulat…
