Deep reinforcement learning (DRL) is a promising approach for developing legged locomotion skills. However, the iterative design process that is inevitable in practice is poorly supported by the default methodology. It is difficult to predict the outcomes of changes made to the reward functions, policy architectures, and the set of tasks being trained on. In this paper, we propose a practical method that allows the reward function to be fully redefined on each successive design iteration while limiting the deviation from the previous iteration. We characterize policies via sets of Deterministic Action Stochastic State (DASS) tuples, which represent the deterministic policy state-action pairs as sampled from the states visited by the trained stochastic policy. New policies are trained using a policy gradient algorithm which then mixes RL-based policy gradients with gradient updates defined by the DASS tuples. The tuples also allow for robust policy distillation to new network architectures. We demonstrate the effectiveness of this iterative-design approach on the bipedal robot Cassie, achieving stable walking with different gait styles at various speeds. We demonstrate the successful transfer of policies learned in simulation to the physical robot without any dynamics randomization, and that variable-speed walking policies for the physical robot can be represented by a small dataset of 5-10k tuples.
Beyond Inverted Pendulums: Task-optimal Simple Models of Legged Locomotion
University of Pennsylvania（宾夕法尼亚大学）；Shanghai Jiao Tong University（上海交通大学）；UM-SJTU Joint Institute（上海交通大学密西根大学联合学院）；General Robotics, Automation, Sensing and Perception (GRASP) Laboratory（通用机器人、自动化、感知与操作实验室（GRASP实验室））
Facebook（脸书）；Facebook AI Research（脸书人工智能研究院）；University of California, Berkeley（加州大学伯克利分校）
Learning to locomote to arbitrary goals on hardware remains a challenging problem for reinforcement learning. In this paper, we present a hierarchical learning framework that improves sample-efficiency and generalizability of locomotion skills on real-world robots. Our approach divides the problem of goal-oriented locomotion into two sub-problems: learning diverse primitives skills, and using model-based planning to sequence these skills. We parametrize our primitives as cyclic movements, improving sample-efficiency of learning on a 18 degrees of freedom robot. Then, we learn coarse dynamics m…
State-of-the-art approaches to legged locomotion are widely dependent on the use of models like the linear inverted pendulum (LIP) and the spring-loaded inverted pendulum (SLIP), popular because their simplicity enables a wide array of tools for planning, control, and analysis. However, they inevitably limit the ability to execute complex tasks or agile maneuvers. In this work, we aim to automatically synthesize models that remain low-dimensional but retain the capabilities of the high-dimensional system. For example, if one were to restore a small degree of complexity to LIP, SLIP, or a simil…
This study proposes an integrated task and motion planning method for dynamic locomotion in partially observable environments with multi-level safety guarantees. This layered planning framework is composed of a high-level symbolic task planner and a low-level phase-space motion planner. A belief abstraction at the task planning level enables belief estimation of dynamic obstacle locations and guarantees navigation safety with collision avoidance. The high-level task planner, i.e., a two-level navigation planner, employs linear temporal logic for a reactive game synthesis between the robot and …
The ability to realize nonlinear controllers with formal guarantees on dynamic robotic systems has the potential to enable more complex robotic behaviors -- yet, realizing these controllers is often practically challenging. To address this challenge, this paper presents the end-to-end realization of dynamic bipedal locomotion on an underactuated bipedal robot via hybrid zero dynamics and control Lyapunov functions. A compliant model of Cassie is represented as a hybrid system to set the stage for a trajectory optimization framework. With the goal of achieving a variety of walking speeds in all…
MOVE: Multi-skill Omnidirectional Legged Locomotion with Limited View in 3D Environments
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院 cyber-system 研究所）；State Key Laboratory of Industrial Control Technology（工业控制技术国家重点实验室）
Collaborative Robotics and Intelligent Systems Institute, Oregon State University（俄勒冈州立大学协作机器人与智能系统研究所）；Agility Robotics
Recent work has demonstrated the success of reinforcement learning (RL) for training bipedal locomotion policies for real robots. This prior work, however, has focused on learning joint-coordination controllers based on an objective of following joint trajectories produced by already available controllers. As such, it is difficult to train these approaches to achieve higher-level goals of legged locomotion, such as simply specifying the desired end-effector foot movement or ground reaction forces. In this work, we propose an approach for integrating knowledge of the robot system into RL to all…
This paper presents a gait controller for bipedal robots to achieve highly agile walking over various terrains given local slope and friction cone information. Without these considerations, untimely impacts can cause a robot to trip and inadequate tangential reaction forces at the stance foot can cause slippages. We address these challenges by combining, in a novel manner, a model based on an Angular Momentum Linear Inverted Pendulum (ALIP) and a Model Predictive Control (MPC) foot placement planner that is executed by the method of virtual constraints. The process starts with abstracting from…
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院 cyber-system 研究所）
Achieving versatile robot locomotion requires motor skills which can adapt to previously unseen situations. We propose a Multi-Expert Learning Architecture (MELA) that learns to generate adaptive skills from a group of representative expert skills. During training, MELA is first initialised by a distinct set of pre-trained experts, each in a separate deep neural network (DNN). Then by learning the combination of these DNNs using a Gating Neural Network (GNN), MELA can acquire more specialised experts and transitional skills across various locomotion modes. During runtime, MELA constantly blend…
MA-ROESL: Motion-aware Rapid Reward Optimization for Efficient Robot Skill Learning from Single Videos
College of Information Science and Technology, Eastern Institute of Technology（信息科学与技术学院，东部技术学院）；School of Computer Science and Technology, The University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；Department of Computing, The Hong Kong Polytechnic University（计算学院，香港理工大学）
University of Edinburgh（爱丁堡大学）；University College London（伦敦大学学院）
This work developed a kernel-based residual learning framework for quadrupedal robotic locomotion. Initially, a kernel neural network is trained with data collected from an MPC controller. Alongside a frozen kernel network, a residual controller network is trained via reinforcement learning to acquire generalized locomotion skills and resilience against external perturbations. With this proposed framework, a robust quadrupedal locomotion controller is learned with high sample efficiency and controllability, providing omnidirectional locomotion at continuous velocities. Its versatility and robu…
The common approach for local navigation on challenging environments with legged robots requires path planning, path following and locomotion, which usually requires a locomotion control policy that accurately tracks a commanded velocity. However, by breaking down the navigation problem into these sub-tasks, we limit the robot's capabilities since the individual tasks do not consider the full solution space. In this work, we propose to solve the complete problem by training an end-to-end policy with deep reinforcement learning. Instead of continuously tracking a precomputed path, the robot nee…
Robots operating in human environments need various skills, like slow and fast walking, turning, side-stepping, and many more. However, building robot controllers that can exhibit such a large range of behaviors is a challenging problem that requires tedious investigation for every task. We present a unified model-based control algorithm for imitating different animal gaits without expensive simulation training or real-world fine-tuning. Our method consists of stance and swing leg controllers using a centroidal dynamics model augmented with online adaptation techniques. We also develop a whole…
Deep reinforcement learning produces robust locomotion policies for legged robots over challenging terrains. To date, few studies have leveraged model-based methods to combine these locomotion skills with the precise control of manipulators. Here, we incorporate external dynamics plans into learning-based locomotion policies for mobile manipulation. We train the base policy by applying a random wrench sequence on the robot base in simulation and adding the noisified wrench sequence prediction to the policy observations. The policy then learns to counteract the partially-known future disturbanc…
This study proposes a hierarchically integrated framework for safe task and motion planning (TAMP) of bipedal locomotion in a partially observable environment with dynamic obstacles and uneven terrain. The high-level task planner employs linear temporal logic (LTL) for a reactive game synthesis between the robot and its environment and provides a formal guarantee on navigation safety and task completion. To address environmental partial observability, a belief abstraction is employed at the high-level navigation planner to estimate the dynamic obstacles' location. Accordingly, a synthesized ac…
Efficient Learning of A Unified Policy For Whole-body Manipulation and Locomotion Skills
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学智能系统与控制研究院）；State Key Laboratory of Industrial Control Technology of Zhejiang University（浙江大学工业控制技术状态重点实验室）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
University of British Columbia（不列颠哥伦比亚大学）；Oregon State University（俄勒冈州立大学）
Bipedal locomotion skills are challenging to develop. Control strategies often use local linearization of the dynamics in conjunction with reduced-order abstractions to yield tractable solutions. In these model-based control strategies, the controller is often not fully aware of many details, including torque limits, joint limits, and other non-linearities that are necessarily excluded from the control computations for simplicity. Deep reinforcement learning (DRL) offers a promising model-free approach for controlling bipedal locomotion which can more fully exploit the dynamics. However, curre…
FAIR；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Hierarchical learning has been successful at learning generalizable locomotion skills on walking robots in a sample-efficient manner. However, the low-dimensional "latent" action used to communicate between two layers of the hierarchy is typically user-designed. In this work, we present a fully-learned hierarchical framework, that is capable of jointly learning the low-level controller and the high-level latent action space. Once this latent space is learned, we plan over continuous latent actions in a model-predictive control fashion, using a learned high-level dynamics model. This framework …
Multi-Loco: Unifying Multi-Embodiment Legged Locomotion via Reinforcement Learning Augmented Diffusion
Southern University of Science and Technology, China（南方科技大学）；University of Notre Dame, United States（Notre Dame 大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute, China（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）；LimX Dynamics, Shenzhen, China（LimX 动力学（深圳））
Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning
Georgia Institute of Technology（佐治亚理工学院）；The University of Southern California（南加州大学）；Technische Universität München（慕尼黑技术大学）；Google DeepMind（谷歌DeepMind）；The AI Institute（人工智能研究所）；The Institute for Human and Machine Cognition（人机认知研究所）；Duke University（杜克大学）；Kuwait University（科威特大学）；Stanford University（斯坦福大学）；CNRS-University of Montpellier（国家科学研究中心-蒙彼利埃大学）；LIRMM CNRS-AIST Joint Robotics Laboratory（联合机器人实验室）；Simon Fraser University（西蒙弗雷泽大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA（英伟达）；Carnegie Mellon University（卡内基梅隆大学）；Harbin Institute of Technology（哈尔滨工业大学）
University of Notre Dame（圣母大学）；Robotics at Google（谷歌机器人部门）
Reinforcement Learning (RL) has witnessed great strides for quadruped locomotion, with continued progress in the reliable sim-to-real transfer of policies. However, it remains a challenge to reuse a policy on another robot, which could save time for retraining. In this work, we present a framework for zero-shot policy retargeting wherein diverse motor skills can be transferred between robots of different shapes and sizes. The new framework centers on a planning-and-control pipeline that systematically integrates RL and Model Predictive Control (MPC). The planning stage employs RL to generate a…
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；School of Design（设计学院）；ZJU-University of Illinois Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）；Zhejiang University（浙江大学）；System Design and Intelligent Manufacturing（系统设计与智能制造学院）；Design & Learning Research Group（设计与学习研究组）
The Cassie bipedal robot designed by Agility Robotics is providing academics a common platform for sharing and comparing algorithms for locomotion, perception, and navigation. This paper focuses on feedback control for standing and walking using the methods of virtual constraints and gait libraries. The designed controller was implemented six weeks after the robot arrived at the University of Michigan and allowed it to stand in place as well as walk over sidewalks, grass, snow, sand, and burning brush. The controller for standing also enables the robot to ride a Segway. A model of the Cassie r…
RPG: Robust Policy Gating for Smooth Multi-Skill Transitions in Humanoid Fighting
Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University（人工智能与机器人中心，深圳国际研究生院，清华大学）；Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Shanghai Jiao Tong University（上海交通大学）
Developing robust walking controllers for bipedal robots is a challenging endeavor. Traditional model-based locomotion controllers require simplifying assumptions and careful modelling; any small errors can result in unstable control. To address these challenges for bipedal locomotion, we present a model-free reinforcement learning framework for training robust locomotion policies in simulation, which can then be transferred to a real bipedal Cassie robot. To facilitate sim-to-real transfer, domain randomization is used to encourage the policies to learn behaviors that are robust across variat…
Motivated towards achieving multi-modal locomotion, in this paper, we develop a framework for a bipedal robot to dynamically ride a pair of Hovershoes over various terrain. Our developed control strategy enables the Cassie bipedal robot to interact with the Hovershoes to balance, regulate forward and rotational velocities, achieve fast turns, and move over flat terrain, slopes, stairs, and rough outdoor terrain. Our sensor suite comprising of tracking and depth cameras for visual SLAM as well as our Dijkstra-based global planner and timed elastic band-based local planning framework enables us …
ETH Zürich（苏黎世联邦理工学院）；Delft University of Technology（代尔夫特理工大学）；Boston Dynamics（波士顿动力公司）
Wheeled-legged robots have the potential for highly agile and versatile locomotion. The combination of legs and wheels might be a solution for any real-world application requiring rapid, and long-distance mobility skills on challenging terrain. In this paper, we present an online trajectory optimization framework for wheeled quadrupedal robots capable of executing hybrid walking-driving locomotion strategies. By breaking down the optimization problem into a wheel and base trajectory planning, locomotion planning for high dimensional wheeled-legged robots becomes more tractable, can be solved i…
