Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
ICube Lab Robotics Department Strasbourg University UMR 7357 CNRS（斯特拉斯堡大学ICube实验室机器人系 UMR 7357 CNRS）；ICube Lab UMR 7357 Centre National de la Recherche Scientifique CNRS（ICube实验室 UMR 7357 法国国家科学研究中心）
Rongrong Liu, Florent Nageotte, Philippe Zanne, Michel de Mathelin, Birgitta Dresp-Langley
Deep learning has provided new ways of manipulating, processing and analyzing data. It sometimes may achieve results comparable to, or surpassing human expert performance, and has become a source of inspiration in the era of artificial intelligence. Another subfield of machine learning named reinforcement learning, tries to find an optimal behavior strategy through interactions with the environment. Combining deep learning and reinforcement learning permits resolving critical issues relative to the dimensionality and scalability of data in tasks with sparse reward signals, such as robotic manipulation and control tasks, that neither method permits resolving when applied on its own. In this paper, we present recent significant progress of deep reinforcement learning algorithms, which try to tackle the problems for the application in the domain of robotic manipulation control, such as sample efficiency and generalization. Despite these continuous improvements, currently, the challenges of learning robust and versatile manipulation skills for robots with deep reinforcement learning are still far from being resolved for real world applications.
We study how choices of input point cloud coordinate frames impact learning of manipulation skills from 3D point clouds. There exist a variety of coordinate frame choices to normalize captured robot-object-interaction point clouds. We find that different frames have a profound effect on agent learning performance, and the trend is similar across 3D backbone networks. In particular, the end-effector frame and the target-part frame achieve higher training efficiency than the commonly used world frame and robot-base frame in many tasks, intuitively because they provide helpful alignments among po…
SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation
The University of Tokyo（东京大学）；ByteDance Seed（字节跳动 Seed）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）
This paper presents ADAMANT, a set of software modules that provides grasp planning capabilities to an existing robot planning and control software framework. Our presented work allows a user to adapt a manipulation task to be used under widely different scenarios with minimal user input, thus reducing the operator's cognitive load. The developed tools include (1) plugin-based components that make it easy to extend default capabilities and to use third-party grasp libraries, (2) An object-centric way to define task constraints, (3) A user-friendly Rviz interface to use the grasp planner utilit…
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））；University of Aberdeen（阿伯丁大学）
Robot learning of real-world manipulation tasks remains challenging and time consuming, even though actions are often simplified by single-step manipulation primitives. In order to compensate the removed time dependency, we additionally learn an image-to-image transition model that is able to predict a next state including its uncertainty. We apply this approach to bin picking, the task of emptying a bin using grasping as well as pre-grasping manipulation as fast as possible. The transition model is trained with up to 42000 pairs of real-world images before and after a manipulation action. Our…
MoTo: A Zero-shot Plug-in Interaction-aware Navigation for General Mobile Manipulation
Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Beijing Key Laboratory of Embodied Intelligence Systems（北京市具身智能系统重点实验室）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Nanyang Technological University（南洋理工大学）
In this work, we present a manipulation planning algorithm for a robot to keep an object stable under changing external forces. We particularly focus on the case where a human may be applying forceful operations, e.g. cutting or drilling, on an object that the robot is holding. The planner produces an efficient plan by intelligently deciding when the robot should change its grasp on the object as the human applies the forces. The planner also tries to choose subsequent grasps such that they will minimize the number of regrasps that will be required in the long-term. Furthermore, as it switches…
The problem of optimizing a sequence of tasks for a robot, also known as multi-point manufacturing, is a well-studied problem. Many of these solutions use a variant of the Traveling Salesman Problem (TSP) and seek to find the minimum distance or time solution. Optimal solution methods struggle to run in real-time and scale for larger problems. In online planning applications where the tasks being executed are fast, the computational time to optimize the ordering can dominate the total execution time. The optimal solution in this application is defined as the computational time for planning plu…
Recent progress in robotic manipulation has dealt with the case of previously unknown objects in the context of relatively simple tasks, such as bin-picking. Existing methods for more constrained problems, however, such as deliberate placement in a tight region, depend more critically on shape information to achieve safe execution. This work deals with pick-and-constrained placement of objects without access to geometric models. The objective is to pick an object and place it safely inside a desired goal region without any collisions, while minimizing the time and the sensing operations requir…
Georgia Institute of Technology（佐治亚理工学院）；Georgia Tech Research Institute（佐治亚理工研究院）
Perception is an essential part of robotic manipulation in a semi-structured environment. Traditional approaches produce a narrow task-specific prediction (e.g., object's 6D pose), that cannot be adapted to other tasks and is ill-suited for deformable objects. In this paper, we propose using canonical mapping as a near-universal and flexible object descriptor. We demonstrate that common object representations can be derived from a single pre-trained canonical mapping model, which in turn can be generated with minimal manual effort using an automated data generation and training pipeline. We pe…
Precision robotic manipulation tasks (insertion, screwing, precisely pick, precisely place) are required in many scenarios. Previous methods achieved good performance on such manipulation tasks. However, such methods typically require tedious calibration or expensive sensors. 3D/RGB-D cameras and torque/force sensors add to the cost of the robotic application and may not always be economical. In this work, we aim to solve these but using only weak-calibrated and low-cost webcams. We propose Binocular Alignment Learning (BAL), which could automatically learn the eye-hand coordination and points…
SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation
CFCS, School of Computer Science, Peking University（北京大学计算机学院CFCS）；Institute for AI, Peking University（北京大学人工智能研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；PKU-WUHAN Institute for Artificial Intelligence, China（北京大学武汉人工智能研究院）
Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition
Research Center for Intelligent Robotics, School of Astronautics, Northwestern Polytechnical University（西北工业大学航天学院智能机器人研究中心）；National Key Laboratory of Aerospace Flight Dynamics, Northwestern Polytechnical University（西北工业大学航空航天飞行动力学全国重点实验室）
Despite decades of research, general purpose in-hand manipulation remains one of the unsolved challenges of robotics. One of the contributing factors that limit current robotic manipulation systems is the difficulty of precisely sensing contact forces -- sensing and reasoning about contact forces are crucial to accurately control interactions with the environment. As a step towards enabling better robotic manipulation, we introduce DIGIT, an inexpensive, compact, and high-resolution tactile sensor geared towards in-hand manipulation. DIGIT improves upon past vision-based tactile sensors by min…
Technical University of Munich（慕尼黑工业大学）；ABB Corporate Research（ABB企业研究院）；German Aerospace Center (DLR)（德国航空航天中心）
In this letter, we present an approach for learning in-hand manipulation skills with a low-cost, underactuated prosthetic hand in the presence of irreversible events. Our approach combines reinforcement learning based on visual perception with low-level reactive control based on tactile perception, which aims to avoid slipping. The objective of the reinforcement learning level consists not only in fulfilling the in-hand manipulation goal, but also in minimizing the intervention of the tactile reactive control. This way, the occurrence of object slipping during the learning procedure, which we …
As robots gain capabilities to enter our human-centric world, they require formalism and algorithms that enable smart and efficient interactions. This is challenging, especially for robotic manipulators with complex tasks that may require collaboration with humans. Prior works approach this problem through reactive synthesis and generate strategies for the robot that guarantee task completion by assuming an adversarial human. While this assumption gives a sound solution, it leads to an "unfriendly" robot that is agnostic to the human intentions. We relax this assumption by formulating the prob…
Recent studies on quadruped robots have focused on either locomotion or mobile manipulation using a robotic arm. Legged robots can manipulate heavier and larger objects using non-prehensile manipulation primitives, such as planar pushing, to drive the object to the desired location. In this paper, we present a novel hierarchical model predictive control (MPC) for contact optimization of the manipulation task. Using two cascading MPCs, we split the loco-manipulation problem into two parts: the first to optimize both contact force and contact location between the robot and the object, and the se…
We present a planning and control framework for physics-based manipulation under uncertainty. The key idea is to interleave robust open-loop execution with closed-loop control. We derive robustness metrics through contraction theory. We use these metrics to plan trajectories that are robust to both state uncertainty and model inaccuracies. However, fully robust trajectories are extremely difficult to find or may not exist for many multi-contact manipulation problems. We separate a trajectory into robust and non-robust segments through a minimum cost path search on a robustness graph. Robust se…
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-巴塞罗那自治大学工业机器人与信息技术研究所）
In this paper we present a Deep Reinforcement Learning approach to solve dynamic cloth manipulation tasks. Differing from the case of rigid objects, we stress that the followed trajectory (including speed and acceleration) has a decisive influence on the final state of cloth, which can greatly vary even if the positions reached by the grasped points are the same. We explore how goal positions for non-grasped points can be attained through learning adequate trajectories for the grasped points. Our approach uses few demonstrations to improve control policy learning, and a sparse reward approach …
