Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
Google Research, Robotics at Google Team（谷歌研究院，谷歌机器人团队）；University of Southern California（南加州大学）；University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）
Ryan Julian, Benjamin Swanson, Gaurav S. Sukhatme, Sergey Levine, Chelsea Finn, Karol Hausman
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment. Can we efficiently adapt previously learned behaviors to new environments, objects and percepts in the real world? In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption. In particular, we demonstrate how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning, including changes in background, object shape and appearance, lighting conditions, and robot morphology. Further, this adaptation uses less than 0.2% of the data necessary to learn the task from scratch. We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning, and that pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data. We also find that these positive results hold in a limited continual learning setting, in which we repeatedly fine-tune a single lineage of policies using data from a succession of new tasks. Our empirical conclusions are consistently supported by experiments on simulated manipulation tasks, and by 52 unique fine-tuning experiments on a real robotic grasping system pre-trained on 580,000 grasps.
Georgia Institute of Technology（佐治亚理工学院）；Twentieth Century Fox（二十世纪福克斯公司）；Starfleet Academy（星际舰队学院）
Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly pred…
Learning robotic tasks in the real world is still highly challenging and effective practical solutions remain to be found. Traditional methods used in this area are imitation learning and reinforcement learning, but they both have limitations when applied to real robots. Combining reinforcement learning with pre-collected demonstrations is a promising approach that can help in learning control policies to solve robotic tasks. In this paper, we propose an algorithm that uses novel techniques to leverage offline expert data using offline and online training to obtain faster convergence and impro…
Many robotic tasks involving some form of 3D visual perception greatly benefit from a complete knowledge of the working environment. However, robots often have to tackle unstructured environments and their onboard visual sensors can only provide incomplete information due to limited workspaces, clutter or object self-occlusion. In recent years, deep learning architectures for shape completion have begun taking traction as effective means of inferring a complete 3D object representation from partial visual data. Nevertheless, most of the existing state-of-the-art approaches provide a fixed outp…
Robotic manipulation of deformable objects gains great attention due to its wide applications including medical surgery, home assistance, and automatic food preparation. The ability to deform soft objects remains a great challenge for robots due to difficulties in defining the problem mathematically. In this paper, we address the problem of shaping a piece of dough-like deformable material into a 2D target shape presented upfront. We use a 6 degree-of-freedom WidowX-250 Robot Arm equipped with a rolling pin and information collected from an RGB-D camera and a tactile sensor. We present and com…
Intuitive Surgical Inc.（直觉外科公司）；Massachusetts Institute of Technology（麻省理工学院）；University of Bern（伯尔尼大学）；University of Alberta（阿尔伯塔大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Washington（华盛顿大学）；NVIDIA GmbH.（英伟达有限公司）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) UCL（伦敦大学学院韦尔科姆/工程和物理科学研究委员会介入与外科科学中心）；Lyft Inc.（来福车公司）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Beijing Institute of Technology（北京理工大学）；Technical University of Munich（慕尼黑工业大学）；German Cancer Research Center（德国癌症研究中心）；Kings College London（伦敦国王学院）
In mainstream computer vision and machine learning, public datasets such as ImageNet, COCO and KITTI have helped drive enormous improvements by enabling researchers to understand the strengths and limitations of different algorithms via performance comparison. However, this type of approach has had limited translation to problems in robotic assisted surgery as this field has never established the same level of common datasets and benchmarking methods. In 2015 a sub-challenge was introduced at the EndoVis workshop where a set of robotic images were provided with automatically generated annotati…
Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Université de Lorraine（洛林大学）
The most data-efficient algorithms for reinforcement learning in robotics are model-based policy search algorithms, which alternate between learning a dynamical model of the robot and optimizing a policy to maximize the expected return given the model and its uncertainties. Among the few proposed approaches, the recently introduced Black-DROPS algorithm exploits a black-box optimization algorithm to achieve both high data-efficiency and good computation times when several cores are used; nevertheless, like all model-based policy search approaches, Black-DROPS does not scale to high dimensional…
Recent advances in deep reinforcement learning (RL) have demonstrated its potential to learn complex robotic manipulation tasks. However, RL still requires the robot to collect a large amount of real-world experience. To address this problem, recent works have proposed learning from expert demonstrations (LfD), particularly via inverse reinforcement learning (IRL), given its ability to achieve robust performance with only a small number of expert demonstrations. Nevertheless, deploying IRL on real robots is still challenging due to the large number of robot experiences it requires. This paper …
University of Pennsylvania（宾夕法尼亚大学）；Stanford University（斯坦福大学）；UC Berkeley（加州大学伯克利分校）
Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works, and then use this learned model to plan coordinated sequences of actions to bring about desired outcomes. However, learning a model that captures the dynamics of complex skills represents a major challenge: if the agent needs a good model to perform these skills, it might never be able to collect the experience on its own that is required to learn these delicate and complex behaviors. Instead, we can imagine augmenting the training set with observational data of othe…
A Haptic Robot Finger Designed for Guqin Instrument Playing
The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong - Shenzhen（香港中文大学（深圳））；School of Physics and Optoelectronic Engineering, Shenzhen University（深圳大学物理与光电工程学院）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen（移动网络与移动多媒体技术国家重点实验室（深圳））；ZTE Corporation（中兴通讯）；Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）
Data augmentation can be a simple yet powerful tool for autonomous robots to fully utilise available data for selfsupervised identification of atypical scenes or objects. State-of-the-art augmentation methods arbitrarily embed "structural" peculiarity on typical images so that classifying these artefacts can provide guidance for learning representations for the detection of anomalous visual signals. In this paper, however, we argue that learning such structure-sensitive representations can be a suboptimal approach to some classes of anomaly (e.g., unhealthy fruits) which could be better recogn…
Southern University of Science and Technology（南方科技大学）
Many researchers have identified robotics as a potential solution to the aging population faced by many developed and developing countries. If so, how should we address the cognitive acceptance and ambient control of elderly assistive robots through design? In this paper, we proposed an explorative design of an ambient SuperLimb (Supernumerary Robotic Limb) system that involves a pneumatically-driven robotic cane for at-home motion assistance, an inflatable vest for compliant human-robot interaction, and a depth sensor for ambient intention detection. The proposed system aims at providing acti…
Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Robotics
1 Learning \& Adaptive Systems Group, Department of Computer Science, ETH Zurich, Switzerland. 2 Dynamic Systems Lab, Institute for Aerospace Studies, University of Toronto, Canada.
