Reproducing the diverse and agile locomotion skills of animals has been a longstanding challenge in robotics. While manually-designed controllers have been able to emulate many complex behaviors, building such controllers involves a time-consuming and difficult development process, often requiring substantial expertise of the nuances of each skill. Reinforcement learning provides an appealing alternative for automating the manual effort involved in the development of controllers. However, designing learning objectives that elicit the desired behaviors from an agent can also require a great deal of skill-specific expertise. In this work, we present an imitation learning system that enables legged robots to learn agile locomotion skills by imitating real-world animals. We show that by leveraging reference motion data, a single learning-based approach is able to automatically synthesize controllers for a diverse repertoire behaviors for legged robots. By incorporating sample efficient domain adaptation techniques into the training process, our system is able to learn adaptive policies in simulation that can then be quickly adapted for real-world deployment. To demonstrate the effectiveness of our system, we train an 18-DoF quadruped robot to perform a variety of agile behaviors ranging from different locomotion gaits to dynamic hops and turns.
Dynamic quadruped locomotion over challenging terrains with precise foot placements is a hard problem for both optimal control methods and Reinforcement Learning (RL). Non-linear solvers can produce coordinated constraint satisfying motions, but often take too long to converge for online application. RL methods can learn dynamic reactive controllers but require carefully tuned shaping rewards to produce good gaits and can have trouble discovering precise coordinated movements. Imitation learning circumvents this problem and has been used with motion capture data to extract quadruped gaits for …
BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）
Integrating Learning-Based Manipulation and Physics-Based Locomotion for Whole-Body Badminton Robot Control
School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；DeepCode Robotics（DeepCode机器人公司）；Department of Sports, Shanghai Jiao Tong University（上海交通大学体育学院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
Learning agile skills is one of the main challenges in robotics. To this end, reinforcement learning approaches have achieved impressive results. These methods require explicit task information in terms of a reward function or an expert that can be queried in simulation to provide a target control output, which limits their applicability. In this work, we propose a generative adversarial method for inferring reward functions from partial and potentially physically incompatible demonstrations for successful skill acquirement where reference or expert demonstrations are not easily accessible. Mo…
SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience
LAAS-CNRS（法国国家科学研究中心自动化与系统分析实验室）；Université de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）；New York University（纽约大学）；Artificial and Natural Intelligence Toulouse Institute（图卢兹人工与自然智能研究所）
Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion
Department of Computer Science, Technical University of Darmstadt（德累斯顿技术大学计算机科学系）；MAB Robtics；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI）系统AI机器人学习研究部）；Robotics Institute Germany (RIG)（德国机器人研究所）
Neural control of memory-constrained, agile robots requires small, yet highly performant models. We leverage graph hyper networks to learn graph hyper policies trained with off-policy reinforcement learning resulting in networks that are two orders of magnitude smaller than commonly used networks yet encode policies comparable to those encoded by much larger networks trained on the same task. We show that our method can be appended to any off-policy reinforcement learning algorithm, without any change in hyperparameters, by showing results across locomotion and manipulation tasks. Further, we …
Institute for Robotics and Intelligent Machines（机器人与智能机器研究所）；School of Electrical and Computer Engineering（电气与计算机工程学院）；Georgia Institute of Technology（佐治亚理工学院）
We present an end-to-end imitation learning system for agile, off-road autonomous driving using only low-cost sensors. By imitating a model predictive controller equipped with advanced sensors, we train a deep neural network control policy to map raw, high-dimensional observations to continuous steering and throttle commands. Compared with recent approaches to similar tasks, our method requires neither state estimation nor on-the-fly planning to navigate the vehicle. Our approach relies on, and experimentally validates, recent imitation learning theory. Empirically, we show that policies train…
Animation and machine learning research have shown great advancements in the past decade, leading to robust and powerful methods for learning complex physically-based animations. However, learning can take hours or days, especially if no reference movement data is available. In this paper, we propose and evaluate a novel combination of techniques for accelerating the learning of stable locomotion movements through self-imitation learning of synthetic animations. First, we produce synthetic and cyclic reference movement using a recent online tree search approach that can discover stable walking…
Massachusetts Institute of Technology（麻省理工学院）；NSF AI Institute for Artificial Intelligence and Fundamental Interactions（美国国家科学基金会人工智能与基础交互研究所）
Agile maneuvers such as sprinting and high-speed turning in the wild are challenging for legged robots. We present an end-to-end learned controller that achieves record agility for the MIT Mini Cheetah, sustaining speeds up to 3.9 m/s. This system runs and turns fast on natural terrains like grass, ice, and gravel and responds robustly to disturbances. Our controller is a neural network trained in simulation via reinforcement learning and transferred to the real world. The two key components are (i) an adaptive curriculum on velocity commands and (ii) an online system identification strategy f…
Legged robots pose one of the greatest challenges in robotics. Dynamic and agile maneuvers of animals cannot be imitated by existing methods that are crafted by humans. A compelling alternative is reinforcement learning, which requires minimal craftsmanship and promotes the natural evolution of a control policy. However, so far, reinforcement learning research for legged robots is mainly limited to simulation, and only few and comparably simple examples have been deployed on real systems. The primary reason is that training with real robots, particularly with dynamically balancing systems, is …
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Vanderbilt University（范德堡大学）
We tackle the problem of perceptive locomotion in dynamic environments. In this problem, a quadrupedal robot must exhibit robust and agile walking behaviors in response to environmental clutter and moving obstacles. We present a hierarchical learning framework, named PRELUDE, which decomposes the problem of perceptive locomotion into high-level decision-making to predict navigation commands and low-level gait generation to realize the target commands. In this framework, we train the high-level navigation controller with imitation learning on human demonstrations collected on a steerable cart a…
Jump-Start Reinforcement Learning with Self-Evolving Priors for Extreme Monopedal Locomotion
State Key Laboratory of Intelligent Green Vehicle and Mobility, School of Vehicle and Mobility, Tsinghua University, Beijing, China（智能绿色车辆与移动 state key laboratory，车辆与移动学院，清华大学，北京，中国）；College of AI, Tsinghua University, Beijing, China（人工智能学院，清华大学，北京，中国）；SunrisingAI Ltd., Beijing, China（SunrisingAI有限公司，北京，中国）
University of Edinburgh（爱丁堡大学）；University College London（伦敦大学学院）
This work developed a kernel-based residual learning framework for quadrupedal robotic locomotion. Initially, a kernel neural network is trained with data collected from an MPC controller. Alongside a frozen kernel network, a residual controller network is trained via reinforcement learning to acquire generalized locomotion skills and resilience against external perturbations. With this proposed framework, a robust quadrupedal locomotion controller is learned with high sample efficiency and controllability, providing omnidirectional locomotion at continuous velocities. Its versatility and robu…
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Tsinghua University（清华大学）
Developing robust vision-guided controllers for quadrupedal robots in complex environments, with various obstacles, dynamical surroundings and uneven terrains, is very challenging. While Reinforcement Learning (RL) provides a promising paradigm for agile locomotion skills with vision inputs in simulation, it is still very challenging to deploy the RL policy in the real world. Our key insight is that aside from the discrepancy in the domain gap, in visual appearance between the simulation and the real world, the latency from the control pipeline is also a major cause of difficulty. In this pape…
An attached arm can significantly increase the applicability of legged robots to several mobile manipulation tasks that are not possible for the wheeled or tracked counterparts. The standard hierarchical control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion. However, this is ineffective. It requires immense engineering to support coordination between the arm and legs, and error can propagate across modules causing non-smooth unnatural motions. It is also biological implausible given evidence for strong motor synergies across limbs.…
Robotics at Google（谷歌机器人）；Georgia Institute of Technology（佐治亚理工学院）
Developing agile behaviors for legged robots remains a challenging problem. While deep reinforcement learning is a promising approach, learning truly agile behaviors typically requires tedious reward shaping and careful curriculum design. We formulate agile locomotion as a multi-stage learning problem in which a mentor guides the agent throughout the training. The mentor is optimized to place a checkpoint to guide the movement of the robot's center of mass while the student (i.e. the robot) learns to reach these checkpoints. Once the student can solve the task, we teach the student to perform …
The success of popular algorithms for deep reinforcement learning, such as policy-gradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difficultly in credit assignment. Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, …
Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-…
Evaluation of an Actuated Spine in Agile Quadruped Locomotion
Department of Computer Science, Technical University of Darmstadt（德意志技术大学达姆施塔特分校计算机科学系）；IDEAS Research Institute（IDEAS研究所）；Institute of Robotics and Machine Intelligence, Poznan University of Technology（波兹南技术大学机器人与机器智能研究所）
Developing controllers for agile locomotion is a long-standing challenge for legged robots. Reinforcement learning (RL) and Evolution Strategy (ES) hold the promise of automating the design process of such controllers. However, dedicated and careful human effort is required to design training environments to promote agility. In this paper, we present a multi-agent learning system, in which a quadruped robot (protagonist) learns to chase another robot (adversary) while the latter learns to escape. We find that this adversarial training process not only encourages agile behaviors but also effect…
University of Aveiro（阿威罗大学）；University of Porto（波尔图大学）
Humanoid robots are made to resemble humans but their locomotion abilities are far from ours in terms of agility and versatility. When humans walk on complex terrains, or face external disturbances, they combine a set of strategies, unconsciously and efficiently, to regain stability. This paper tackles the problem of developing a robust omnidirectional walking framework, which is able to generate versatile and agile locomotion on complex terrains. The Linear Inverted Pendulum Model and Central Pattern Generator concepts are used to develop a closed-loop walk engine, which is then combined with…
Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer t…
TAGA: Terrain-aware Active Gaze Learning for Generalizable Agile Humanoid Locomotion
MarmotLab, National University of Singapore（马尔莫实验室，新加坡国立大学）；Center of X-Mechanics, Zhejiang University（浙大X力学中心）；South China University of Technology（华南理工大学）
Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion
Department of Intelligence and Information, Graduate School of Convergence Science and Technology, Seoul National University（智能信息系，融合科学与技术研究院，首尔国立大学）；X Technologies（1X技术公司）；Department of Intelligence and Information, Graduate School of Convergence Science and Technology, ASRI, AIIS, Seoul National University, Republic of Korea, and Advanced Institute of Convergence Technology (AICT), Suwon, Republic of Korea（智能信息系，融合科学与技术研究院，ASRI、AIIS、首尔国立大学，韩国，以及融合技术研究所（AICT），Suwon，韩国）
Collaborative Robotics and Intelligent Systems Institute, Oregon State University（俄勒冈州立大学协作机器人与智能系统研究所）
The complex dynamics of agile robotic legged locomotion requires motion planning to intelligently adjust footstep locations. Often, bipedal footstep and motion planning use mathematically simple models such as the linear inverted pendulum, instead of dynamically-rich models that do not have closed-form solutions. We propose a real-time optimization method to plan for dynamical models that do not have closed form solutions and experience irrecoverable failure. Our method uses a data-driven approximation of the step-to-step dynamics and of a failure margin function. This failure margin function …
Diffusion-based learning of contact plans for agile locomotion
Munich Institute of Robotics and Machine Intelligence, Technical University of Munich, Germany（慕尼黑机器人与机器智能研究所，慕尼黑技术大学）；Tandon School of Engineering, New York University, USA（纽约大学土曼工学院）；Max-Planck Institute for Intelligent Systems, Tübingen, Germany（马克斯·普朗克智能系统研究所，图宾根）
Agile and Generalized Legged Locomotion via Attention-Based Neural Map Encoding
Robotic Systems Lab, ETH Zurich, Switzerland（机器人系统实验室，苏黎世联邦理工学院，瑞士）；Secure, Reliable, and Intelligent Systems Lab, ETH Zurich, Switzerland（安全、可靠和智能系统实验室，苏黎世联邦理工学院，瑞士）；ETH AI Center, Switzerland（苏黎世联邦理工学院人工智能中心，瑞士）
Institute for Robotics and Intelligent Machines, Georgia Institute of Technology（佐治亚理工学院机器人与智能机器研究所）；MIT Biomimetic Robotics Lab（麻省理工学院仿生机器人实验室）；NAVER LABS（NAVER实验室）
In this letter, we present a versatile hierarchical offline planning algorithm, along with an online control pipeline for agile quadrupedal locomotion. Our offline planner alternates between optimizing centroidal dynamics for a reduced-order model and whole-body trajectory optimization, with the aim of achieving dynamics consensus. Our novel momentum-inertia-aware centroidal optimization, which uses an equimomental ellipsoid parameterization, is able to generate highly acrobatic motions via ``inertia shaping". Our whole-body optimization approach significantly improves upon the quality of stan…
Walk the PLANC: Physics-Guided RL for Agile Humanoid Locomotion on Constrained Footholds
Department of Mechanical and Civil Engineering, California Institute of Technology（机械与土木工程系，加州理工学院）；Department of Computing and Mathematical Sciences, California Institute of Technology（计算与数学科学系，加州理工学院）
ATRos: Learning Energy-Efficient Agile Locomotion for Wheeled-legged Robots
Shanghai Huawei Technologies Co., Ltd.（上海华为技术有限公司）；College of Future Information Technology, Fudan University（复旦大学未来信息科技学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；School of Aeronautic and Astronautics, Zhejiang University（浙江大学航空宇航学院）；School of Electronics and Communication Engineering, Sun Yat-sen University（中山大学电子与通信工程学院）
