Simulation-to-real transfer is an important strategy for making reinforcement learning practical with real robots. Successful sim-to-real transfer systems have difficulty producing policies which generalize across tasks, despite training for thousands of hours equivalent real robot time. To address this shortcoming, we present a novel approach to efficiently learning new robotic skills directly on a real robot, based on model-predictive control (MPC) and an algorithm for learning task representations. In short, we show how to reuse the simulation from the pre-training step of sim-to-real methods as a tool for foresight, allowing the sim-to-real policy adapt to unseen tasks. Rather than end-to-end learning policies for single tasks and attempting to transfer them, we first use simulation to simultaneously learn (1) a continuous parameterization (i.e. a skill embedding or latent) of task-appropriate primitive skills, and (2) a single policy for these skills which is conditioned on this representation. We then directly transfer our multi-skill policy to a real robot, and actuate the robot by choosing sequences of skill latents which actuate the policy, with each latent corresponding to a pre-learned primitive skill controller. We complete unseen tasks by choosing new sequences of skill latents to control the robot using MPC, where our MPC model is composed of the pre-trained skill policy executed in the simulation environment, run in parallel with the real robot. We discuss the background and principles of our method, detail its practical implementation, and evaluate its performance by using our method to train a real Sawyer Robot to achieve motion tasks such as drawing and block pushing.
Six-DoF Stewart Platform Motion Simulator Control using Switchable Model Predictive Control
School of Information Engineering, Pingdingshan University（平顶山学院信息工程学院）；School of Aerospace, Transport and Manufacturing, Cranfield University（克兰菲尔德大学航空航天、运输与制造学院）；College of Civil Aviation, Nanjing University of Aeronautics and Astronautics（南京航空航天大学民航学院）
A task-sequencing simulator in robotics manipulation to integrate simulation-for-learning and simulation-for-execution is introduced. Unlike existing machine-learning simulation where a non-decomposed simulation is used to simulate a training scenario, the task-sequencing simulator runs a composed simulation using building blocks. This way, the simulation-for-learning is structured similarly to a multi-step simulation-for-execution. To compose both learning and execution scenarios, a unified trainable-and-composable description of blocks called a concept model is proposed and used. Using the s…
Georgia Institute of Technology（佐治亚理工学院）；Honda Research Institute USA, Inc.（本田美国研究院）
It can be difficult to autonomously produce driver behavior so that it appears natural to other traffic participants. Through Inverse Reinforcement Learning (IRL), we can automate this process by learning the underlying reward function from human demonstrations. We propose a new IRL algorithm that learns a goal-conditioned spatiotemporal reward function. The resulting costmap is used by Model Predictive Controllers (MPCs) to perform a task without any hand-designing or hand-tuning of the cost function. We evaluate our proposed Goal-conditioned SpatioTemporal Zeroing Maximum Entropy Deep IRL (G…
Amirkabir University of Technology (Tehran Polytechnic)（阿米尔卡比尔理工大学（德黑兰理工大学））；Sharif University of Technology（谢里夫理工大学）
Haptic training simulators generally consist of three major components, namely a human operator, a haptic interface, and a virtual environment. Appropriate dynamic modeling of each of these components can have far-reaching implications for the whole system's performance improvement in terms of transparency, the analogy to the real environment, and stability. In this paper, we developed a virtual-based haptic training simulator for Endoscopic Sinus Surgery (ESS) by doing a dynamic characterization of the phenomenological sinus tissue fracture in the virtual environment, using an input-constrain…
Online reinforcement learning (RL) algorithms are often difficult to deploy in complex human-facing applications as they may learn slowly and have poor early performance. To address this, we introduce a practical algorithm for incorporating human insight to speed learning. Our algorithm, Constraint Sampling Reinforcement Learning (CSRL), incorporates prior domain knowledge as constraints/restrictions on the RL policy. It takes in multiple potential policy constraints to maintain robustness to misspecification of individual constraints while leveraging helpful ones to learn quickly. Given a bas…
Model-free Reinforcement Learning (RL) works well when experience can be collected cheaply and model-based RL is effective when system dynamics can be modeled accurately. However, both assumptions can be violated in real world problems such as robotics, where querying the system can be expensive and real-world dynamics can be difficult to model. In contrast to RL, Model Predictive Control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics. MPC performance is usually limited by fa…
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）；Tampere University of technology（坦佩雷理工大学）
In this paper, we propose a new model predictive control (MPC) formulation for autonomous driving. The novelty of our MPC stems from the following results. Firstly, we adopt an alternating minimization approach wherein linear velocities and angular accelerations are alternately optimized. We show that in contrast to the joint optimization, the alternating minimization exploits the structure of the problem better, which in turn translates to reduction in computation time. Secondly, our MPC explicitly incorporates the time dependent non-linear actuator dynamics that captures the transient respon…
A Multi-Simulation Approach with Model Predictive Control for Anafi Drones
University of Stuttgart, Faculty of Aerospace Engineering and Geodesy, Institute of Flight Mechanics and Control (iFR), Flight Robotics and Perception Group (FRPG)（斯图加特大学航空航天工程与大地测量学院飞行力学与控制研究所飞行机器人与感知组）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）
Real-World Deployment of Massively Parallel Sampling-Based MPC for Contact-Rich Manipulation
Intelligent Autonomous Systems Lab, TU Darmstadt（达姆施塔特工业大学智能自主系统实验室）；Interactive Robot Perception & Learning Lab, TU Darmstadt（达姆施塔特工业大学交互式机器人感知与学习实验室）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Robotics Institute Germany (RIG)（德国机器人研究所）；College of Engineering and Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）
Do My Samples Help? Compute Budget Allocation for Real-World Sampling-Based MPC in Contact-Rich Manipulation
TU Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Robotics Institute Germany (RIG)（德国机器人研究所）；School of Computing, DePaul University（德保罗大学计算学院）
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Resolving edge-cases in autonomous driving, head-to-head autonomous racing is getting a lot of attention from the industry and academia. In this study, we propose a game-theoretic model predictive control (MPC) approach for head-to-head autonomous racing and data-driven model identification method. For the practical estimation of nonlinear model parameters, we adopted the hyperband algorithm, which is used for neural model training in machine learning. The proposed controller comprises three modules: 1) game-based opponents' trajectory predictor, 2) high-level race strategy planner, and 3) MPC…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
This paper investigates the vision-based autonomous driving with deep learning and reinforcement learning methods. Different from the end-to-end learning method, our method breaks the vision-based lateral control system down into a perception module and a control module. The perception module which is based on a multi-task learning neural network first takes a driver-view image as its input and predicts the track features. The control module which is based on reinforcement learning then makes a control decision based on these features. In order to improve the data efficiency, we propose visual…
Technical University of Munich（慕尼黑工业大学）；Delft University of Technology（代尔夫特理工大学）
Autonomous cars can reduce road traffic accidents and provide a safer mode of transport. However, key technical challenges, such as safe navigation in complex urban environments, need to be addressed before deploying these vehicles on the market. Teleoperation can help smooth the transition from human operated to fully autonomous vehicles since it still has human in the loop providing the scope of fallback on driver. This paper presents an Active Safety System (ASS) approach for teleoperated driving. The proposed approach helps the operator ensure the safety of the vehicle in complex environme…
Birla Institute of Technology and Science, Pilani（皮拉尼比特拉科技学院）
Legged robots can traverse challenging terrain, use perception to plan their safe foothold positions, and navigate the environment. Such unique mobility capabilities make these platforms a perfect candidate for scenarios such as search and rescue, inspection, and exploration tasks. While traversing through such terrains, the robot's instability is a significant concern. Many times the robot needs to switch gaits depending on its environment. Due to the complex dynamics of quadruped robots, classical PID control fails to provide high stability. Thus, there is a need for advanced control methods…
Manipal Institute of Technology（马尼帕尔理工学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Motion Planning under uncertainty is critical for safe self-driving. In this paper, we propose a unified obstacle avoidance framework that deals with 1) uncertainty in ego-vehicle motion; and 2) prediction uncertainty of dynamic obstacles from environment. A two-stage traffic participant trajectory predictor comprising short-term and long-term prediction is used in the planning layer to generate safe but not over-conservative trajectories for the ego vehicle. The prediction module cooperates well with existing planning approaches. Our work showcases its effectiveness in a Frenet frame planner.…
HPTune: Hierarchical Proactive Tuning for Collision-Free Model Predictive Control
The University of Hong Kong（香港大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of New South Wales（新南威尔士大学）
Chance-Aware Lane Change with High-Level Model Predictive Control Through Curriculum Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；The Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Model-free policy learning has enabled robust performance of complex tasks with relatively simple algorithms. However, this simplicity comes at the cost of requiring an Oracle and arguably very poor sample complexity. This renders such methods unsuitable for physical systems. Variants of model-based methods address this problem through the use of simulators, however, this gives rise to the problem of policy transfer from simulated to the physical system. Model mismatch due to systematic parameter shift and unmodelled dynamics error may cause sub-optimal or unsafe behavior upon direct transfer.…
Hybrid Control Policy for Artificial Pancreas via Ensemble Deep Reinforcement Learning
East China University of Science and Technology（华东理工大学）；Shanghai Jiao Tong University Affiliated Sixth People’s Hospital（上海交通大学附属第六人民医院）；Shanghai Diabetes Institute（上海市糖尿病研究所）；Shanghai Clinical Center for Diabetes（上海市糖尿病临床医学中心）
School of Informatics, The University of Edinburgh（爱丁堡大学信息学院）；Shenzhen Institute for Artificial Intelligence and Robotics for Society (AIRS)（深圳市人工智能与机器人研究院）；Oxford Robotics Institute, University of Oxford（牛津大学牛津机器人研究所）
In this paper, we study a wheeled robot with a prismatic extension joint. This allows the robot to build up momentum to perform jumps over obstacles and to swing up to the upright position after the loss of balance. We propose a template model for the class of such two-wheeled jumping robots. This model can be considered as the simplest wheeled-legged system. We provide an analytical derivation of the system dynamics which we use inside a model predictive controller (MPC). We study the behavior of the model and demonstrate highly dynamic motions such as swing-up and jumping. Furthermore, these…
Tel-Aviv University（特拉维夫大学）；Technion（以色列理工学院）；University of California, Berkeley（加州大学伯克利分校）
To compute robust 2D assembly plans, we present an approach that combines geometric planning with a deep neural network. We train the network using the Box2D physics simulator with added stochastic noise to yield robustness scores--the success probabilities of planned assembly motions. As running a simulation for every assembly motion is impractical, we train a convolutional neural network to map assembly operations, given as an image pair of the subassemblies before and after they are mated, to a robustness score. The neural network prediction is used within a planner to quickly prune out mot…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Programming languages, libraries, and development tools have transformed the application development processes for mobile computing and machine learning. This paper introduces the CyPhyHouse - a toolchain that aims to provide similar programming, debugging, and deployment benefits for distributed mobile robotic applications. Users can develop hardware-agnostic, distributed applications using the high-level, event driven Koord programming language, without requiring expertise in controller design or distributed network protocols. The modular, platform-independent middleware of CyPhyHouse implem…
University of California San Diego（加州大学圣地亚哥分校）；Hong Kong University of Science and Technology（香港科技大学）
Recent advancements in reinforcement learning algorithms have opened doors for researchers to operate and optimize building energy management systems autonomously. However, the lack of an easily configurable building dynamical model and energy management task simulation and evaluation platform has arguably slowed the progress in developing advanced and dedicated reinforcement learning (RL) and control algorithms for building operation tasks. Here we propose "BEAR", a physics-principled Building Environment for Control And Reinforcement Learning. The platform allows researchers to benchmark bot…
Neural-Rendezvous: Provably Robust Guidance and Control to Encounter Interstellar Objects
Jet Propulsion Laboratory（喷气推进实验室）；California Institute of Technology（加州理工学院）；National Aeronautics and Space Administration（美国国家航空航天局）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
