One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion
Department of Computer Science, Technical University of Darmstadt（德累斯顿技术大学计算机科学系）；Institute of Robotics and Machine Intelligence, Poznan University of Technology（波兹南技术大学机器人与机器智能研究所）；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI）系统AI for机器人学习研究部）；IDEAS NCBR, Warsaw, Poland（华沙波兰IDEAS NCBR）；Centre for Cognitive Science（认知科学中心）
Nico Bohlinger, Grzegorz Czechmanowski, Maciej Krupka, Piotr Kicki, Krzysztof Walas, Jan Peters, Davide Tateo
Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion. While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments. We introduce URMA, the Unified Robot Morphology Architecture, to close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a potential first step in building a foundation model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.
Towards Embodiment Scaling Laws in Robot Locomotion
University of California San Diego（加州大学圣地亚哥分校）；Hillbot Inc（Hillbot公司）；Cornell University（康奈尔大学）；Technical University of Darmstadt（达姆施塔特技术大学）；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI）；德国机器人研究所；hessian.AI）；Robotics Institute Germany；hessian.AI
Multi-Loco: Unifying Multi-Embodiment Legged Locomotion via Reinforcement Learning Augmented Diffusion
Southern University of Science and Technology, China（南方科技大学）；University of Notre Dame, United States（Notre Dame 大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute, China（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）；LimX Dynamics, Shenzhen, China（LimX 动力学（深圳））
Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion
Department of Computer Science, Technical University of Darmstadt（德累斯顿技术大学计算机科学系）；MAB Robtics；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI）系统AI机器人学习研究部）；Robotics Institute Germany (RIG)（德国机器人研究所）
PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning
Department of Computer Science & Technology, Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University（计算机科学与技术系、人工智能研究所、BNRist中心、清华-博世联合机器学习中心、THBI实验室、清华大学）
We derive a hybrid feedback control law for the lateral leg spring (LLS) model so that the center of mass of a legged runner follows a curved path in horizontal plane. The control law enables the runner to change the placement and the elasticity of its legs to move in a desired direction. Stable motion along a curved path is achieved using curvature, bearing and relative distance between the runner and the curve as feedback. Constraints on leg parameters determine the class of curves that can be followed. We also derive an optimal control law that stabilizes the orientation of the runner's bod…
Rapid Embodiment Adaptation for Quadrupedal Locomotion
UC San Diego（加州大学圣迭戈分校）；Stanford University（斯坦福大学）；TU Darmstadt（达姆施塔特工业大学）；Sudo AI GmbH（Sudo AI 有限公司）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Robotics Institute Germany（德国机器人研究所）
Model-Based Reinforcement Learning Exploits Passive Body Dynamics for High-Performance Biped Robot Locomotion
Department of Mechanical Science and Bioengineering, Graduate School of Engineering and Science, The University of Osaka（大阪大学机械科学与生物工程学系）；Department of Electrical and Mechanical Engineering, Nagoya Institute of Technology（名古屋工业大学电气机械学系）
Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually loo…
Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
Aalto University（阿alto大学）；University of Edinburgh（爱丁堡大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；Deep Render；Imperial College London（伦敦帝国理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；CIFAR AI Chair（CIFAR人工智能主席）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；University of Oulu（奥卢大学）
Optimal control is a successful approach to generate motions for complex robots, in particular for legged locomotion. However, these techniques are often too slow to run in real time for model predictive control or one needs to drastically simplify the dynamics model. In this work, we present a method to learn to predict the gradient and hessian of the problem value function, enabling fast resolution of the predictive control problem with a one-step quadratic program. In addition, our method is able to satisfy constraints like friction cones and unilateral constraints, which are important for …
UniPilot: Enabling GPS-Denied Autonomy Across Embodiments
Department of Engineering Cybernetics, O. S. Bragstads Plass 2D, Norwegian University of Science and Technology (NTNU), Trondheim, Norway（工程 cybernetics 系，挪威科学技术大学（NTNU），特隆赫姆，挪威）
Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hierarchical framework to automatically decompose complex locomotion tasks. A high-level policy issues commands in a latent space and also selects for how long the low-level policy will execute the latent command. Concurrently, the low-level policy uses the latent command and only the robot's on-board sensors to control the robot's actuators. Our approach allows the high-level policy to run at a lower frequency than the…
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院 cyber-system 研究所）
Achieving versatile robot locomotion requires motor skills which can adapt to previously unseen situations. We propose a Multi-Expert Learning Architecture (MELA) that learns to generate adaptive skills from a group of representative expert skills. During training, MELA is first initialised by a distinct set of pre-trained experts, each in a separate deep neural network (DNN). Then by learning the combination of these DNNs using a Gating Neural Network (GNN), MELA can acquire more specialised experts and transitional skills across various locomotion modes. During runtime, MELA constantly blend…
Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning
Georgia Institute of Technology（佐治亚理工学院）；The University of Southern California（南加州大学）；Technische Universität München（慕尼黑技术大学）；Google DeepMind（谷歌DeepMind）；The AI Institute（人工智能研究所）；The Institute for Human and Machine Cognition（人机认知研究所）；Duke University（杜克大学）；Kuwait University（科威特大学）；Stanford University（斯坦福大学）；CNRS-University of Montpellier（国家科学研究中心-蒙彼利埃大学）；LIRMM CNRS-AIST Joint Robotics Laboratory（联合机器人实验室）；Simon Fraser University（西蒙弗雷泽大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA（英伟达）；Carnegie Mellon University（卡内基梅隆大学）；Harbin Institute of Technology（哈尔滨工业大学）
UC San Diego（加州大学圣地亚哥分校）；Tsinghua University（清华大学）；UC Berkeley（加州大学伯克利分校）
We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) with a Transformer-based model that learns to combine proprioceptive information and high-dimensional depth sensor inputs. While learning-based locomotion has made great advances using RL, most methods still rely on domain randomization for training blind agents that generalize to challenging terrains. Our key insight is that proprioceptive states only offer contact measurements for immediate reaction, whereas an agent equipped with visual sensory observations can learn to proactively maneuver environments wit…
Southern University of Science and Technology（南方科技大学）；Massachusetts Institute of Technology（麻省理工学院）；University of British Columbia（不列颠哥伦比亚大学）；National University of Singapore（新加坡国立大学）
Reinforcement learning (RL) is a popular data-driven method that has demonstrated great success in robotics. Previous works usually focus on learning an end-to-end (direct) policy to directly output joint torques. While the direct policy seems convenient, the resultant performance may not meet our expectations. To improve its performance, more sophisticated reward functions or more structured policies can be utilized. This paper focuses on the latter because the structured policy is more intuitive and can inherit insights from previous model-based controllers. It is unsurprising that the struc…
Massachusetts Institute of Technology（麻省理工学院）；NSF AI Institute for Artificial Intelligence and Fundamental Interactions（美国国家科学基金会人工智能与基础交互研究所）
Agile maneuvers such as sprinting and high-speed turning in the wild are challenging for legged robots. We present an end-to-end learned controller that achieves record agility for the MIT Mini Cheetah, sustaining speeds up to 3.9 m/s. This system runs and turns fast on natural terrains like grass, ice, and gravel and responds robustly to disturbances. Our controller is a neural network trained in simulation via reinforcement learning and transferred to the real world. The two key components are (i) an adaptive curriculum on velocity commands and (ii) an online system identification strategy f…
Royal Veterinary College（皇家兽医学院）；DeepMind（深度思维）；University of California, Irvine（加州大学欧文分校）；Imperial College London（帝国理工学院）
Muscle-actuated control is a research topic that spans multiple domains, including biomechanics, neuroscience, reinforcement learning, robotics, and graphics. This type of control is particularly challenging as bodies are often overactuated and dynamics are delayed and non-linear. It is however a very well tested and tuned actuation mechanism that has undergone millions of years of evolution with interesting properties exploiting passive forces and efficient energy storage of muscle-tendon units. To facilitate research on muscle-actuated simulation, we release a 3D musculoskeletal simulation o…
In this paper, we propose a locomotion training framework where a control policy and a state estimator are trained concurrently. The framework consists of a policy network which outputs the desired joint positions and a state estimation network which outputs estimates of the robot's states such as the base linear velocity, foot height, and contact probability. We exploit a fast simulation environment to train the networks and the trained networks are transferred to the real robot. The trained policy and state estimator are capable of traversing diverse terrains such as a hill, slippery plate, …
Convex model predictive controls (MPCs) with a single rigid body model have demonstrated strong performance on real legged robots. However, convex MPCs are limited by their assumptions such as small rotation angle and pre-defined gait, limiting the richness of potential solutions. We remove those assumptions and solve the complete mixed-integer non-convex programming with single rigid body model. We first collect datasets of pre-solved problems offline, then learn the problem-solution map to solve this optimization fast for MPC. If warm-starts can be found, offline problems can be solved close…
Istituto Italiano di Tecnologia（意大利理工学院）；University of Genoa（热那亚大学）；University of Manchester（曼彻斯特大学）
This paper presents a Non-Linear Model Predictive Controller for humanoid robot locomotion with online step adjustment capabilities. The proposed controller considers the Centroidal Dynamics of the system to compute the desired contact forces and torques and contact locations. Differently from bipedal walking architectures based on simplified models, the presented approach considers the reduced centroidal model, thus allowing the robot to perform highly dynamic movements while keeping the control problem still treatable online. We show that the proposed controller can automatically adjust the …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；KU Leuven（鲁汶大学）
Most legged robots are built with leg structures from serially mounted links and actuators and are controlled through complex controllers and sensor feedback. In comparison, animals developed multi-segment legs, mechanical coupling between joints, and multi-segmented feet. They run agile over all terrains, arguably with simpler locomotion control. Here we focus on developing foot mechanisms that resist slipping and sinking also in natural terrain. We present first results of multi-segment feet mounted to a bird-inspired robot leg with multi-joint mechanical tendon coupling. Our one- and two-se…
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Tsinghua University（清华大学）
Developing robust vision-guided controllers for quadrupedal robots in complex environments, with various obstacles, dynamical surroundings and uneven terrains, is very challenging. While Reinforcement Learning (RL) provides a promising paradigm for agile locomotion skills with vision inputs in simulation, it is still very challenging to deploy the RL policy in the real world. Our key insight is that aside from the discrepancy in the domain gap, in visual appearance between the simulation and the real world, the latency from the control pipeline is also a major cause of difficulty. In this pape…
Enforcing safety in the presence of stochastic uncertainty is a challenging problem. Traditionally, researchers have proposed safety in the statistical mean as a safety measure in this case. However, ensuring safety in the statistical mean is only reasonable if system's safe behavior in the large number of runs is of interest, which precludes the use of mean safety in practical scenarios. In this paper, we propose a risk sensitive notion of safety called conditional-value-at-risk (CVaR) safety, which is concerned with safe performance in the worst case realizations. We introduce CVaR barrier f…
Southern University of Science and Technology（南方科技大学）
This paper studies the problem of constructing polytopic representations of planar regions from depth camera readings. This problem is of great importance for terrain mapping in complicated environment and has great potentials in legged locomotion applications. To address the polytopic planar region characterization problem, we propose a two-stage solution scheme. At the first stage, the planar regions embedded within a sequence of depth images are extracted individually first and then merged to establish a terrain map containing only planar regions in a selected frame. To simplify the represe…
The University of Texas at Austin（得克萨斯大学奥斯汀分校）；US Army Research Laboratory（美国陆军研究实验室）；Sony AI（索尼人工智能）
While Adversarial Imitation Learning (AIL) algorithms have recently led to state-of-the-art results on various imitation learning benchmarks, it is unclear as to what impact various design decisions have on performance. To this end, we present here an organizing, modular framework called Reinforcement-learning-based Adversarial Imitation Learning (RAIL) that encompasses and generalizes a popular subclass of existing AIL approaches. Using the view espoused by RAIL, we create two new IfO (Imitation from Observation) algorithms, which we term SAIfO: SAC-based Adversarial Imitation from Observatio…
Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-…
Locomotion is a crucial challenge for legged robots that is addressed "effortlessly" by biological networks abundant in nature, named central pattern generators (CPG). The multitude of CPG network models that have so far become biomimetic robotic controllers is not applicable to the emerging neuromorphic hardware, depriving mobile robots of a robust walking mechanism that would result in inherently energy-efficient systems. Here, we propose a brain-morphic CPG controler based on a comprehensive spiking neural-astrocytic network that generates two gait patterns for a hexapod robot. Building on …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Laboratory for Analysis and Architecture of Systems, Centre National pour la Recherche Scientifique（法国国家科学研究中心系统分析与架构实验室）
We present a new open-source torque-controlled legged robot system, with a low-cost and low-complexity actuator module at its core. It consists of a high-torque brushless DC motor and a low-gear-ratio transmission suitable for impedance and force control. We also present a novel foot contact sensor suitable for legged locomotion with hard impacts. A 2.2 kg quadruped robot with a large range of motion is assembled from eight identical actuator modules and four lower legs with foot contact sensors. Leveraging standard plastic 3D printing and off-the-shelf parts results in a lightweight and inexp…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）；University of Stuttgart（斯图加特大学）；KU Leuven（鲁汶大学）
Animals run robustly in diverse terrain. This locomotion robustness is puzzling because axon conduction velocity is limited to a few ten meters per second. If reflex loops deliver sensory information with significant delays, one would expect a destabilizing effect on sensorimotor control. Hence, an alternative explanation describes a hierarchical structure of low-level adaptive mechanics and high-level sensorimotor control to help mitigate the effects of transmission delays. Motivated by the concept of an adaptive mechanism triggering an immediate response, we developed a tunable physical damp…
Spring-mass models are well established tools for the analysis and control of legged locomotion. Among the alternatives, spring-loaded inverted pendulum (SLIP) model has shown to be a very accurate descriptor of animal locomotion. Despite its wide use, the SLIP model includes non-integrable stance dynamics that prevent analytical solutions for its equations of motion. Fortunately, there are approximate analytical solutions for different SLIP variants. However, the practicality of such approximations are mostly tested on simulation studies with a few notable exceptions. This thesis extends upon…
