Recently, work on reinforcement learning (RL) for bipedal robots has successfully learned controllers for a variety of dynamic gaits with robust sim-to-real demonstrations. In order to maintain balance, the learned controllers have full freedom of where to place the feet, resulting in highly robust gaits. In the real world however, the environment will often impose constraints on the feasible footstep locations, typically identified by perception systems. Unfortunately, most demonstrated RL controllers on bipedal robots do not allow for specifying and responding to such constraints. This missing control interface greatly limits the real-world application of current RL controllers. In this paper, we aim to maintain the robust and dynamic nature of learned gaits while also respecting footstep constraints imposed externally. We develop an RL formulation for training dynamic gait controllers that can respond to specified touchdown locations. We then successfully demonstrate simulation and sim-to-real performance on the bipedal robot Cassie. In addition, we use supervised learning to induce a transition model for accurately predicting the next touchdown locations that the controller can achieve given the robot's proprioceptive observations. This model paves the way for integrating the learned controller into a full-order robot locomotion planner that robustly satisfies both balance and environmental constraints.
Collaborative Robotics and Intelligent Systems Institute, Oregon State University（俄勒冈州立大学协作机器人与智能系统研究所）
In this work, we propose a learning approach for 3D dynamic bipedal walking when footsteps are constrained to stepping stones. While recent work has shown progress on this problem, real-world demonstrations have been limited to relatively simple open-loop, perception-free scenarios. Our main contribution is a more advanced learning approach that enables real-world demonstrations, using the Cassie robot, of closed-loop dynamic walking over moderately difficult stepping-stone patterns. Our approach first uses reinforcement learning (RL) in simulation to train a controller that maps footstep comm…
Collaborative Robotics and Intelligent Systems Institute, Oregon State University（俄勒冈州立大学协作机器人与智能系统研究所）
For legged robots to match the athletic capabilities of humans and animals, they must not only produce robust periodic walking and running, but also seamlessly switch between nominal locomotion gaits and more specialized transient maneuvers. Despite recent advancements in controls of bipedal robots, there has been little focus on producing highly dynamic behaviors. Recent work utilizing reinforcement learning to produce policies for control of legged robots have demonstrated success in producing robust walking behaviors. However, these learned policies have difficulty expressing a multitude of…
Ohio State University（俄亥俄州立大学）；Southern University of Science and Technology (SUSTech)（南方科技大学）
In this paper, a hierarchical and robust framework for learning bipedal locomotion is presented and successfully implemented on the 3D biped robot Digit built by Agility Robotics. We propose a cascade-structure controller that combines the learning process with intuitive feedback regulations. This design allows the framework to realize robust and stable walking with a reduced-dimension state and action spaces of the policy, significantly simplifying the design and reducing the sampling efficiency of the learning method. The inclusion of feedback regulation into the framework improves the robus…
Developing robust walking controllers for bipedal robots is a challenging endeavor. Traditional model-based locomotion controllers require simplifying assumptions and careful modelling; any small errors can result in unstable control. To address these challenges for bipedal locomotion, we present a model-free reinforcement learning framework for training robust locomotion policies in simulation, which can then be transferred to a real bipedal Cassie robot. To facilitate sim-to-real transfer, domain randomization is used to encourage the policies to learn behaviors that are robust across variat…
ETH Zurich（苏黎世联邦理工学院）；Sony Group Corporation（索尼集团公司）；National University of Singapore（新加坡国立大学）；Nara Institute of Science and Technology（奈良科学技术研究所）；The University of Tokyo（东京大学）
Robust Humanoid Walking on Compliant and Uneven Terrain with Deep Reinforcement Learning
CNRS-AIST JRL (Joint Robotics Laboratory) IRL, National Institute of Advanced Industrial Science and Technology (AIST), Japan（日本国家先进工业科学与技术研究院（AIST））；University of Tsukuba（筑波大学）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
Bipedal walking is one of the most difficult but exciting challenges in robotics. The difficulties arise from the complexity of high-dimensional dynamics, sensing and actuation limitations combined with real-time and computational constraints. Deep Reinforcement Learning (DRL) holds the promise to address these issues by fully exploiting the robot dynamics with minimal craftsmanship. In this paper, we propose a novel DRL approach that enables an agent to learn omnidirectional locomotion for humanoid (bipedal) robots. Notably, the locomotion behaviors are accomplished by a single control policy…
Florida Institute for Human and Machine Cognition（佛罗里达人机认知研究所）
To increase the speed of operation and reduce operator burden, humanoid robots must be able to function autonomously, even in complex, cluttered environments. For this to be possible, they must be able to quickly and efficiently compute desired footsteps to reach a goal. In this work, we present a new A* footstep planner that utilizes a planar region representation of the environment enable footstep planning over rough terrain. To increase the number of available footholds, we present an approach to allow the use of partial footholds during the planning process. The footstep plan solutions are…
Online footstep planning is essential for bipedal walking robots to be able to walk in the presence of disturbances. Until recently this has been achieved by only optimizing the placement of the footstep, keeping the duration of the step constant. In this paper we introduce a footstep planner capable of optimizing footstep placement and timing in real-time by asynchronously combining two optimizers, which we refer to as asynchronous real-time optimization (ARTO). The first optimizer which runs at approximately 25 Hz, utilizes a fourth-order Runge-Kutta (RK4) method to accurately approximate th…
The ability to track a general walking path with specific timing is crucial to the operational safety and reliability of bipedal robots for avoiding dynamic obstacles, such as pedestrians, in complex environments. This paper introduces an online, full-body motion planner that generates the desired impact-aware motion for fully-actuated bipedal robotic walking. The main novelty of the proposed planner lies in its capability of producing desired motions in real-time that respect the discrete impact dynamics and the desired impact timing. To derive the proposed planner, a full-order hybrid dynami…
Imperial College London（帝国理工学院）；ETH Zürich（苏黎世联邦理工学院）
Online footstep planning is essential for bipedal walking robots, allowing them to walk in the presence of disturbances and sensory noise. Most of the literature on the topic has focused on optimizing the footstep placement while keeping the step timing constant. In this work, we introduce a footstep planner capable of optimizing footstep placement and step time online. The proposed planner, consisting of an Interior Point Optimizer (IPOPT) and an optimizer based on Augmented Lagrangian (AL) method with analytical gradient descent, solves the full dynamics of the Linear Inverted Pendulum (LIP)…
We present a sampling-based kinodynamic planning framework for a bipedal robot in complex environments. Unlike other footstep planner which typically plan footstep locations and the biped dynamics in separate steps, we handle both simultaneously. Three advantages of this approach are (1) the ability to differentiate alternate routes while selecting footstep locations based on the temporal duration of the route as determined by the Linear Inverted Pendulum Model dynamics, (2) the ability to perform collision checking through time so that collisions with moving obstacles are prevented without av…
Dynamic walking on bipedal robots has evolved from an idea in science fiction to a practical reality. This is due to continued progress in three key areas: a mathematical understanding of locomotion, the computational ability to encode this mathematics through optimization, and the hardware capable of realizing this understanding in practice. In this context, this review article outlines the end-to-end process of methods which have proven effective in the literature for achieving dynamic walking on bipedal robots. We begin by introducing mathematical models of locomotion, from reduced order mo…
This paper proposes a novel orientation-aware model predictive control (MPC) for dynamic humanoid walking that can plan footstep locations online. Instead of a point-mass model, this work uses the augmented single rigid body model (aSRBM) to enable the MPC to leverage orientation dynamics and stepping strategy within a unified optimization framework. With the footstep location as part of the decision variables in the aSRBM, the MPC can reason about stepping within the kinematic constraints. A task-space controller (TSC) tracks the body pose and swing leg references output from the MPC, while e…
In this paper, we holistically present a Hybrid-Linear Inverted Pendulum (H-LIP) based approach for synthesizing and stabilizing 3D foot-underactuated bipedal walking, with an emphasis on thorough hardware realization. The H-LIP is proposed to capture the essential components of the underactuated and actuated part of the robotic walking. The robot walking gait is then directly synthesized based on the H-LIP. We comprehensively characterize the periodic orbits of the H-LIP and provably derive the stepping stabilization via its step-to-step (S2S) dynamics, which is then utilized to approximate t…
We present a stepping stabilization control that addresses external push disturbances on bipedal walking robots. The stepping control is synthesized based on the step-to-step (S2S) dynamics of the robot that is controlled to have an approximately constant center of mass (COM) height. We first learn a linear S2S dynamics with bounded model discrepancy from the undisturbed walking behaviors of the robot, where the walking step size is taken as the control input to the S2S dynamics. External pushes are then considered as disturbances to the learned S2S (L-S2S) dynamics. We then apply the system-l…
Delft University of Technology（代尔夫特理工大学）；California Institute of Technology（加州理工学院）
Humans are able to negotiate downstep behaviors -- both planned and unplanned -- with remarkable agility and ease. The goal of this paper is to systematically study the translation of this human behavior to bipedal walking robots, even if the morphology is inherently different. Concretely, we begin with human data wherein planned and unplanned downsteps are taken. We analyze this data from the perspective of reduced-order modeling of the human, encoding the center of mass (CoM) kinematics and contact forces, which allows for the translation of these behaviors into the corresponding reduced-ord…
Florida Institute for Human and Machine Cognition（佛罗里达人机认知研究所）；University of West Florida（西佛罗里达大学）
In trying to build humanoid robots that perform useful tasks in a world built for humans, we address the problem of autonomous locomotion. Humanoid robot planning and control algorithms for walking over rough terrain are becoming increasingly capable. At the same time, commercially available depth cameras have been getting more accurate and GPU computing has become a primary tool in AI research. In this paper, we present a newly constructed behavior control system for achieving fast, autonomous, bipedal walking, without pauses or deliberation. We achieve this using a recently published rapid p…
We present a theoretical analysis of a recent whole body motion planning method, the Randomized Possibility Graph, which uses a high-level decomposition of the feasibility constraint manifold in order to rapidly find routes that may lead to a solution. These routes are then examined by lower-level planners to determine feasibility. In this paper, we show that this approach is probabilistically complete for bipedal robots performing quasi-static walking in "semi-unstructured" environments. Furthermore, we show that the decomposition into higher and lower level planners allows for a considerably…
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；University of Michigan（密歇根大学）；Ohio State University（俄亥俄州立大学）；Purdue University（普渡大学）
Controller design for bipedal walking on dynamic rigid surfaces (DRSes), which are rigid surfaces moving in the inertial frame (e.g., ships and airplanes), remains largely uninvestigated. This paper introduces a hierarchical control approach that achieves stable underactuated bipedal robot walking on a horizontally oscillating DRS. The highest layer of our approach is a real-time motion planner that generates desired global behaviors (i.e., the center of mass trajectories and footstep locations) by stabilizing a reduced-order robot model. One key novelty of this layer is the derivation of the …
Bipedal-Walking-Dynamics Model on Granular Terrains
Department of Mechanical and Aerospace Engineering, Rutgers University（罗格斯大学机械与航空航天工程系）；Bridgewater Raritan High School（布里奇沃特-拉里坦高中）；State Key Lab of Fluid Power Transmission and Control and the School of Mechanical Engineering, Zhejiang University（浙江大学流体动力传动与控制国家重点实验室和机械工程学院）
Real-world applications of bipedal robot walking require accurate, real-time state estimation. State estimation for locomotion over dynamic rigid surfaces (DRS), such as elevators, ships, public transport vehicles, and aircraft, remains under-explored, although state estimator designs for stationary rigid surfaces have been extensively studied. Addressing DRS locomotion in state estimation is a challenging problem mainly due to the nonlinear, hybrid nature of walking dynamics, the nonstationary surface-foot contact points, and hardware imperfections (e.g., limited availability, noise, and drif…
In this paper, we propose a multi-domain control parameter learning framework that combines Bayesian Optimization (BO) and Hybrid Zero Dynamics (HZD) for locomotion control of bipedal robots. We leverage BO to learn the control parameters used in the HZD-based controller. The learning process is firstly deployed in simulation to optimize different control parameters for a large repertoire of gaits. Next, to tackle the discrepancy between the simulation and the real world, the learning process is applied on the physical robot to learn for corrections to the control parameters learned in simulat…
Not until recently, robust robot locomotion has been achieved by deep reinforcement learning (DRL). However, for efficient learning of parametrized bipedal walking, developed references are usually required, limiting the performance to that of the references. In this paper, we propose to design an adaptive reward function for imitation learning from the references. The agent is encouraged to mimic the references when its performance is low, while to pursue high performance when it reaches the limit of references. We further demonstrate that developed references can be replaced by low-quality r…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；New York University（纽约大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
In this paper, we propose a novel framework capable of generating various walking and running gaits for bipedal robots. The main goal is to relax the fixed center of mass (CoM) height assumption of the linear inverted pendulum model (LIPM) and generate a wider range of walking and running motions, without a considerable increase in complexity. To do so, we use the concept of virtual constraints in the centroidal space which enables generating motions beyond walking while keeping the complexity at a minimum. By a proper choice of these virtual constraints, we show that we can generate different…
In this paper, previous works on the Model Predictive Control (MPC) and the Divergent Component of Motion (DCM) for bipedal walking control are extended. To this end, we employ a single MPC which uses a combination of Center of Pressure (CoP) manipulation, step adjustment, and Centroidal Moment Pivot (CMP) modulation to design a robust walking controller. Furthermore, we exploit the concept of time-varying DCM to generalize our walking controller for walking in uneven surfaces. Using our scheme, a general and robust walking controller is designed which can be implemented on robots with differe…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
For humanoids to be deployed in demanding situations, such as search and rescue, highly intelligent decision making and proficient sensorimotor skill is expected. A promising solution is to leverage human prowess by interconnecting robot and human via teleoperation. Towards creating seamless operation, this paper presents a dynamic telelocomotion framework that synchronizes the gait of a human pilot with the walking of a bipedal robot. First, we introduce a method to generate a virtual human walking model from the stepping behavior of a human pilot which serves as a reference for the robot to …
The control of bipedal robotic walking remains a challenging problem in the domains of computation and experiment, due to the multi-body dynamics and various sources of uncertainty. In recent years, there has been a rising trend towards model reduction and the design of intuitive controllers to overcome the gap between assumed model and reality. Despite its viability in practical implementation, this local representation of true dynamics naturally indicate limited scalibility towards more dynamical behaviors. With the goal of moving towards increasingly dynamic behaviors, we leverage the detai…
We present a new walking controller based on 3LP, a 3D model of bipedal walking that is composed of three pendulums to simulate falling, swing and torso dynamics. Taking advantage of linear equations and closed-form solutions of 3LP, the proposed controller projects intermediate states of the biped back to the beginning of the phase for which a discrete LQR controller is designed. After the projection, a proper control policy is generated by this LQR controller and used at the intermediate time. The projection controller reacts to disturbances immediately and compared to the discrete LQR contr…
A safety-critical measure of legged locomotion performance is a robot's ability to track its desired time-varying position trajectory in an environment, which is herein termed as "global-position tracking". This paper introduces a nonlinear control approach that achieves asymptotic global-position tracking for three-dimensional (3-D) bipedal robot walking. Designing a global-position tracking controller presents a challenging problem due to the complex hybrid robot model and the time-varying desired global-position trajectory. Towards tackling this problem, the first main contribution is the c…
Bipedal robots are a prime example of systems which exhibit highly nonlinear dynamics, underactuation, and undergo complex dissipative impacts. This paper discusses methods used to overcome a wide variety of uncertainties, with the end result being stable bipedal walking. The principal contribution of this paper is to establish sufficiency conditions for yielding input to state stable (ISS) hybrid periodic orbits, i.e., stable walking gaits under model-based and phase-based uncertainties. In particular, it will be shown formally that exponential input to state stabilization (e-ISS) of the cont…
Indian Institute of Technology (BHU) Varanasi（瓦拉纳西印度理工学院（BHU））；Ohio State University（俄亥俄州立大学）；Indian Institute of Science（印度科学学院）
In this work, we demonstrate robust walking in the bipedal robot Digit on uneven terrains by just learning a single linear policy. In particular, we propose a new control pipeline, wherein the high-level trajectory modulator shapes the end-foot ellipsoidal trajectories, and the low-level gait controller regulates the torso and ankle orientation. The foot-trajectory modulator uses a linear policy and the regulator uses a linear PD control law. As opposed to neural network-based policies, the proposed linear policy has only 13 learnable parameters, thereby not only guaranteeing sample efficient …
National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）；University of Tsukuba（筑波大学）
Deep reinforcement learning (RL) based controllers for legged robots have demonstrated impressive robustness for walking in different environments for several robot platforms. To enable the application of RL policies for humanoid robots in real-world settings, it is crucial to build a system that can achieve robust walking in any direction, on 2D and 3D terrains, and be controllable by a user-command. In this paper, we tackle this problem by learning a policy to follow a given step sequence. The policy is trained with the help of a set of procedurally generated step sequences (also called foot…
Laboratoire des Sciences du Numérique de Nantes (LS2N)（南特数字科学实验室）；CNRS（法国国家科学研究中心）；Ecole Centrale de Nantes（南特中央理工学院）；Université de Nantes（南特大学）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；University of Michigan（密歇根大学）
This paper seeks insight into stabilization mechanisms for periodic walking gaits in 3D bipedal robots. Based on this insight, a control strategy based on virtual constraints, which imposes coordination between joints rather than a temporal evolution, will be proposed for achieving asymptotic convergence toward a periodic motion. For planar bipeds with one degree of underactuation, it is known that a vertical displacement of the center of mass---with downward velocity at the step transition---induces stability of a walking gait. This paper concerns the qualitative extension of this type of pro…
We present a highly reactive controller which enables bipedal robots to blindly walk over various kinds of uneven terrains while resisting pushes. The high level motion planner does fast online optimization for footstep locations and Center of Mass (CoM) height using the decoupled actuated Spring Loaded Inverted Pendulum (aSLIP) model. The decoupled aSLIP model simplifies the original aSLIP with Linear Inverted Pendulum (LIP) dynamics in horizontal states and spring dynamics in the vertical state. The motion planning can be formulated as a discrete-time Model Predictive Control (MPC) and solve…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Amirkabir University of Technology（阿米尔卡比尔理工大学）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）
In this paper, we present a novel two-level variable Horizon Model Predictive Control (VH-MPC) framework for bipedal locomotion. In this framework, the higher level computes the landing location and timing (horizon length) of the swing foot to stabilize the unstable part of the center of mass (CoM) dynamics, using feedback from the CoM states. The lower level takes into account the swing foot dynamics and generates dynamically consistent trajectories for landing at the desired time as close as possible to the desired location. To do that, we use a simplified model of the robot dynamics project…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
The problem of navigating a bipedal robot to a desired destination in various environments is very important. However, it is very difficult to solve the navigation problem in real time because the computation time is very long due to the nature of the biped robot having a high degree of freedom. In order to overcome this, many scientists suggested navigation through the footstep planning. Usually footstep planning use the shortest distance or angles as the objective function based on the A * algorithm. Recently, the energy required for human walking, which is widely used in human dynamics, app…
California Institute of Technology（加州理工学院）；Technion - Israel Institute of Technology（以色列理工学院）
Dynamic bipedal robot locomotion has achieved remarkable success due in part to recent advances in trajectory generation and nonlinear control for stabilization. A key assumption utilized in both theory and experiments is that the robot's stance foot always makes no-slip contact with the ground, including at impacts. This assumption breaks down on slippery low-friction surfaces, as commonly encountered in outdoor terrains, leading to failure and loss of stability. In this work, we extend the theoretical analysis and trajectory optimization to account for stick-slip transitions at point foot co…
Many methods exist for a bipedal robot to keep its balance while walking. In addition to step size and timing, other strategies are possible that influence the stability of the robot without interfering with the target direction and speed of locomotion. This paper introduces a multifaceted feedback controller that uses numerous different feedback mechanisms, collectively termed corrective actions, to stabilise a core keypoint-based gait. The feedback controller is experimentally effective, yet free of any physical model of the robot, very computationally inexpensive, and requires only a single…
University of Tehran（德黑兰大学）；K. N. Toosi University of Technology（图西理工大学）
Available possibilities to prevent a biped robot from falling down in the presence of severe disturbances are mainly Center of Pressure (CoP) modulation, step location and timing adjustment, and angular momentum regulation. In this paper, we aim at designing a walking pattern generator which employs an optimal combination of these tools to generate robust gaits. In this approach, first, the next step location and timing are decided consistent with the commanded walking velocity and based on the Divergent Component of Motion (DCM) measurement. This stage which is done by a very small-size Quadr…
Robust and energy-efficient bipedal locomotion in robotics is still a challenging topic. In order to address issues in this field, we can take inspiration from nature, by studying human locomotion. The Spring-Loaded Inverted Pendulum (SLIP) model has shown to be a good model for this purpose. However, the human musculoskeletal system enables us to actively modulate leg stiffness, for example when walking in rough terrain with irregular and unexpected height variations of the walking surface. This ability of varying leg stiffness is not considered in conventional SLIP-based models, and therefor…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Technische Universität Ilmenau（伊尔默瑙工业大学）
The hybrid zero dynamics control concept for bipedal walking is extended to include a non-instantaneous double support phase. A symmetric robot that consists of five rigid body segments which are connected by four actuated revolute joints is considered. Periodic walking gaits with a constant average walking speed consists of alternating single (SSP) and double support phases (DSP). Hybrid zero dynamics control designs usually assume an instantaneous DSP, which is a severe limitation. The proposed controllers use continuous SSPs and DSPs. Transitions between both phases are modeled as instantan…
Asymptotically Stable Walking of a Five-Link Underactuated 3D Bipedal Robot
CNRS（法国国家科学研究中心）；IRCCyN（南特中央理工学院机器人研究中心）；Nantes Atlantic University（南特大西洋大学）；University of Michigan（密歇根大学）；National Taiwan University of Science and Technology（台湾科技大学）
The ability to generate robust walking gaits on bipedal robots is key to their successful realization on hardware. To this end, this work extends the method of Hybrid Zero Dynamics (HZD) -- which traditionally only accounts for locomotive stability via periodicity constraints under perfect impact events -- through the inclusion of the saltation matrix with a view toward synthesizing robust walking gaits. By jointly minimizing the norm of the extended saltation matrix and the torque of the robot directly in the gait generation process, we demonstrate that the synthesized gaits are more robust t…
Korea Institute of Science and Technology（韩国科学技术研究院）；UNIVERSITY OF SCIENCE AND TECHNOLOGY（科学技术大学）
In this work, the hierarchical control strategy of template-based control for a bipedal robot is described. The axial force of a compliant leg is redirected to a point, called the virtual pivot point (VPP), of a 2D biped robot, which is located above the CoM of the model, to generate a restoring moment for the trunk motion. The resulting behavior of the model would resemble a virtual pendulum rotating around this VPP, thus aiming for an upright trunk during walking. Inspired by this analysis, we propose a new force redirecting method as a controller for robot walking. Then, these key features …
An omnidirectional closed-loop gait based on the direct feedback of orientation deviation estimates is presented in this paper. At the core of the gait is an open-loop central pattern generator. The orientation feedback is derived from a 3D nonlinear attitude estimator, and split into the relevant angular deviations in the sagittal and lateral planes using the concept of fused angles. These angular deviations from expected are used by a number of independent feedback mechanisms, including one that controls timing, to perform stabilising corrective actions. The tuning of the feedback mechanisms…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Technical University of Munich（慕尼黑工业大学）
Legged locomotion in humans is governed by natural dynamics of the human body and neural control. One mechanism that is assumed to contribute to the high efficiency of human walking is the impulsive ankle push-off, which potentially powers the swing leg catapult. However, the mechanics of the human lower leg with its complex muscle-tendon units spanning over single and multiple joints is not yet understood. Legged robots allow testing the interaction between complex leg mechanics, control, and environment in real-world walking gait. We developed a 0.49m tall, 2.2kg anthropomorphic bipedal robo…
