We present a learning algorithm for training a single policy that imitates multiple gaits of a walking robot. To achieve this, we use and extend MPC-Net, which is an Imitation Learning approach guided by Model Predictive Control (MPC). The strategy of MPC-Net differs from many other approaches since its objective is to minimize the control Hamiltonian, which derives from the principle of optimality. To represent the policies, we employ a mixture-of-experts network (MEN) and observe that the performance of a policy improves if each expert of a MEN specializes in controlling exactly one mode of a hybrid system, such as a walking robot. We introduce new loss functions for single- and multi-gait policies to achieve this kind of expert selection behavior. Moreover, we benchmark our algorithm against Behavioral Cloning and the original MPC implementation on various rough terrain scenarios. We validate our approach on hardware and show that a single learned policy can replace its teacher to control multiple gaits.
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Vanderbilt University（范德堡大学）
We tackle the problem of perceptive locomotion in dynamic environments. In this problem, a quadrupedal robot must exhibit robust and agile walking behaviors in response to environmental clutter and moving obstacles. We present a hierarchical learning framework, named PRELUDE, which decomposes the problem of perceptive locomotion into high-level decision-making to predict navigation commands and low-level gait generation to realize the target commands. In this framework, we train the high-level navigation controller with imitation learning on human demonstrations collected on a steerable cart a…
Benchmarking the Full-Order Model Optimization Based Imitation in the Humanoid Robot Reinforcement Learning Walk
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Federal Research Center “Computer Science and Control” of Russian Academy of Sciences（俄罗斯科学院计算机科学与控制联邦研究中心）
Imitation Learning from Human Motion Alone Does Not Guarantee Biomechanically Plausible Gait Kinetics
Joint Department of Biomedical Engineering, University of North Carolina - Chapel Hill（北卡罗来纳大学教堂山分校联合生物医学工程系）；Department of Electrical and Computer Engineering, North Carolina State University（北卡罗来纳州立大学电气与计算机工程系）；School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学电气、计算机与能源工程学院）
This paper systematically decomposes a quadrupedal robot into bipeds to rapidly generate walking gaits and then recomposes these gaits to obtain quadrupedal locomotion. We begin by decomposing the full-order, nonlinear and hybrid dynamics of a three-dimensional quadrupedal robot, including its continuous and discrete dynamics, into two bipedal systems that are subject to external forces. Using the hybrid zero dynamics (HZD) framework, gaits for these bipedal robots can be rapidly generated (on the order of seconds) along with corresponding controllers. The decomposition is achieved in such a w…
Seoul National University（首尔国立大学）；Georgia Institute of Technology（佐治亚理工学院）
A motion-based control interface promises flexible robot operations in dangerous environments by combining user intuitions with the robot's motor capabilities. However, designing a motion interface for non-humanoid robots, such as quadrupeds or hexapods, is not straightforward because different dynamics and control strategies govern their movements. We propose a novel motion control system that allows a human user to operate various motor tasks seamlessly on a quadrupedal robot. We first retarget the captured human motion into the corresponding robot motion with proper semantics using supervis…
Learning Diverse Natural Behaviors for Enhancing the Agility of Quadrupedal Robots
Department of Control Science and Intelligent Engineering（控制科学与智能工程系）；School of Management and Engineering（管理与工程学院）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）
Tsinghua University（清华大学）；Technical University of Munich（慕尼黑工业大学）
Not until recently, robust bipedal locomotion has been achieved through reinforcement learning. However, existing implementations rely heavily on insights and efforts from human experts, which is costly for the iterative design of robot systems. Also, styles of the learned motion are strictly limited to that of the reference. In this paper, we propose a new way to learn bipedal locomotion from a simple sine wave as the reference for foot heights. With the naive human insight that the two feet should be lifted up alternatively and periodically, we experimentally demonstrate on the Cassie robot …
Swimming Under Constraints: A Safe Reinforcement Learning Framework for Quadrupedal Bio-Inspired Propulsion
FSI lab, Westlake University（西交大实验室，西湖大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhejiang University（浙江大学）；Zhongguancun Academy（中关村学院）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
In this letter, the authors propose a two-step approach to evaluate and verify a true system's capacity to satisfy its operational objective. Specifically, whenever the system objective has a quantifiable measure of satisfaction, i.e. a signal temporal logic specification, a barrier function, etc - the authors develop two separate optimization problems solvable via a Bayesian Optimization procedure detailed within. This dual approach has the added benefit of quantifying the Sim2Real Gap between a system simulator and its hardware counterpart. Our contributions are twofold. First, we show repea…
It is often overlooked by roboticists when designing locomotion controllers for their legged machines, that energy consumption plays an important role in selecting the best gaits for locomotion at high speeds or over long distances. The purpose of this study is to examine four similar asymmetrical quadrupedal gaits that are frequently observed in legged animals in nature. To understand how a specific footfall pattern will change the energetics of a legged system, we first developed a full body model of a quadrupedal robot called A1. And for each gait we created a hybrid system with desired foo…
Effect of Gait Design on Proprioceptive Sensing of Terrain Properties in a Quadrupedal Robot
Department of Mechanical Engineering, Massachusetts Institute of Technology（机械工程系，麻省理工学院）；Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California（Ming Hsieh 电子与计算机工程系，南加州大学）；Department of Earth and Environmental Science, University of Pennsylvania（地球与环境科学系，宾夕法尼亚大学）
本文对比四足机器人的Crawl N' Sense与Trot-Walk步态，证明慢速爬行步态能更稳定、准确地感知可变形地形强度及表面结壳破裂。
United States Military Academy（美国军事学院）；Massachusetts Institute of Technology（麻省理工学院）
The Extra Robotic Legs (XRL) system is a robotic augmentation worn by a human operator consisting of two articulated robot legs that walk with the operator and help bear a heavy backpack payload. It is desirable for the Human-XRL quadruped system to walk with the rear legs lead the front by 25% of the gait period, minimizing the energy lost from foot impacts while maximizing balance stability. Unlike quadrupedal robots, the XRL cannot command the human's limbs to coordinate quadrupedal locomotion. Using a pair of Rimless Wheel models, it is shown that the systems coupled with a spring and damp…
Chinese University of Hong Kong（香港中文大学）；Baidu Inc.（百度公司）；Southern University of Science and Technology（南方科技大学）；Shenzhen Research Institute of the Chinese University of Hong Kong（香港中文大学深圳研究院）
Recently reinforcement learning (RL) has emerged as a promising approach for quadrupedal locomotion, which can save the manual effort in conventional approaches such as designing skill-specific controllers. However, due to the complex nonlinear dynamics in quadrupedal robots and reward sparsity, it is still difficult for RL to learn effective gaits from scratch, especially in challenging tasks such as walking over the balance beam. To alleviate such difficulty, we propose a novel RL-based approach that contains an evolutionary foot trajectory generator. Unlike prior methods that use a fixed tr…
MoE-Loco: Mixture of Experts for Multitask Locomotion
IIIS, Tsinghua University, Beijing, China（清华大学信息学院，北京，中国）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）；Massachusetts Institute of Technology, MA, USA（麻省理工学院，马萨诸塞州，美国）
Biomechanical Comparisons Reveal Divergence of Human and Humanoid Gaits
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences（中国科学院宁波材料技术与工程研究所）
Georgia Institute of Technology（佐治亚理工学院）；Meta Platforms, Inc.（Meta平台公司）
Learning various motor skills for quadrupedal robots is a challenging problem that requires careful design of task-specific mathematical models or reward descriptions. In this work, we propose to learn a single capable policy using deep reinforcement learning by imitating a large number of reference motions, including walking, turning, pacing, jumping, sitting, and lying. On top of the existing motion imitation framework, we first carefully design the observation space, the action space, and the reward function to improve the scalability of the learning as well as the robustness of the final p…
Animation and machine learning research have shown great advancements in the past decade, leading to robust and powerful methods for learning complex physically-based animations. However, learning can take hours or days, especially if no reference movement data is available. In this paper, we propose and evaluate a novel combination of techniques for accelerating the learning of stable locomotion movements through self-imitation learning of synthetic animations. First, we produce synthetic and cyclic reference movement using a recent online tree search approach that can discover stable walking…
Robots operating in human environments need various skills, like slow and fast walking, turning, side-stepping, and many more. However, building robot controllers that can exhibit such a large range of behaviors is a challenging problem that requires tedious investigation for every task. We present a unified model-based control algorithm for imitating different animal gaits without expensive simulation training or real-world fine-tuning. Our method consists of stance and swing leg controllers using a centroidal dynamics model augmented with online adaptation techniques. We also develop a whole…
Robotic Systems Lab, ETH Zürich（苏黎世联邦理工学院机器人系统实验室）
This paper addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries which are difficult to model and predict, the need arises to equip them with the capability to generalize well to unforeseen situations. In this work, we propose a novel technique for training neural-network policies for terrain-aware locomotion, which combines state-of-the-art methods for model-based motion planning and reinforcement learning. Our approach is centered on formulating Markov decision processes using the evaluation of dynam…
Learning how to walk is a sophisticated neurological task for most animals. In order to walk, the brain must synthesize multiple cortices, neural circuits, and diverse sensory inputs. Some animals, like humans, imitate surrounding individuals to speed up their learning. When humans watch their peers, visual data is processed through a visual cortex in the brain. This complex problem of imitation-based learning forms associations between visual data and muscle actuation through Central Pattern Generation (CPG). Reproducing this imitation phenomenon on low power, energy-constrained robots that a…
Our paper proposes a model predictive controller as a single-task formulation that simultaneously optimizes wheel and torso motions. This online joint velocity and ground reaction force optimization integrates a kinodynamic model of a wheeled quadrupedal robot. It defines the single rigid body dynamics along with the robot's kinematics while treating the wheels as moving ground contacts. With this approach, we can accurately capture the robot's rolling constraint and dynamics, enabling automatic discovery of hybrid maneuvers without needless motion heuristics. The formulation's generality thro…
Robert Bosch Centre for Cyber Physical Systems（罗伯特·博世网络物理系统中心）；Indian Institute of Science（印度科学学院）
In this paper, with a view toward fast deployment of locomotion gaits in low-cost hardware, we use a linear policy for realizing end-foot trajectories in the quadruped robot, Stoch $2$. In particular, the parameters of the end-foot trajectories are shaped via a linear feedback policy that takes the torso orientation and the terrain slope as inputs. The corresponding desired joint angles are obtained via an inverse kinematics solver and tracked via a PID control law. Augmented Random Search, a model-free and a gradient-free learning algorithm is used to train this linear policy. Simulation resu…
California Institute of Technology（加州理工学院）；Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）
The hybrid zero dynamics (HZD) approach has become a powerful tool for the gait planning and control of bipedal robots. This paper aims to extend the HZD methods to address walking, ambling and trotting behaviors on a quadrupedal robot. We present a framework that systematically generates a wide range of optimal trajectories and then provably stabilizes them for the full-order, nonlinear and hybrid dynamical models of quadrupedal locomotion. The gait planning is addressed through a scalable nonlinear programming using direct collocation and HZD. The controller synthesis for the exponential sta…
Machine learning algorithms have found several applications in the field of robotics and control systems. The control systems community has started to show interest towards several machine learning algorithms from the sub-domains such as supervised learning, imitation learning and reinforcement learning to achieve autonomous control and intelligent decision making. Amongst many complex control problems, stable bipedal walking has been the most challenging problem. In this paper, we present an architecture to design and simulate a planar bipedal walking robot(BWR) using a realistic robotics sim…
DreamWaQ++: Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learning
Urban Robotics Lab., School of Electrical Engineering, KAIST（韩国科学技术院电子工程系城市机器人实验室）；KRAFTON（KRAFTON公司）；URobotics（URobotics公司）；Laboratory for Information and Decision Systems (LIDS), MIT（麻省理工学院信息与决策系统实验室）
In recent years, reinforcement learning (RL) has shown outstanding performance for locomotion control of highly articulated robotic systems. Such approaches typically involve tedious reward function tuning to achieve the desired motion style. Imitation learning approaches such as adversarial motion priors aim to reduce this problem by encouraging a pre-defined motion style. In this work, we present an approach to augment the concept of adversarial motion prior-based RL to allow for multiple, discretely switchable styles. We show that multiple styles and skills can be learned simultaneously wit…
University of California, Berkeley（加州大学伯克利分校）；Meta AI
In this paper, we present a framework rooted in control and planning that enables quadrupedal robots to traverse challenging terrains with discrete footholds using visual feedback. Navigating discrete terrain is challenging for quadrupeds because the motion of the robot can be aperiodic, highly dynamic, and blind for the hind legs of the robot. Additionally, the robot needs to reason over both the feasible footholds as well as robot velocity by speeding up and slowing down at different parts of the terrain. We build an offline library of periodic gaits which span two trotting steps on the robo…
Efficient Learning of Control Policies for Robust Quadruped Bounding using Pretrained Neural Networks
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院）；Intelligent Robot Research Center, Zhejiang Lab（之江实验室智能机器人研究中心）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）；State Key Laboratory of Industrial Control Technology, Zhejiang University（浙江大学工业控制技术国家重点实验室）
