Imitation Learning (IL) holds great promise for enabling agile locomotion in embodied agents. However, many existing locomotion benchmarks primarily focus on simplified toy tasks, often failing to capture the complexity of real-world scenarios and steering research toward unrealistic domains. To advance research in IL for locomotion, we present a novel benchmark designed to facilitate rigorous evaluation and comparison of IL algorithms. This benchmark encompasses a diverse set of environments, including quadrupeds, bipeds, and musculoskeletal human models, each accompanied by comprehensive datasets, such as real noisy motion capture data, ground truth expert data, and ground truth sub-optimal data, enabling evaluation across a spectrum of difficulty levels. To increase the robustness of learned agents, we provide an easy interface for dynamics randomization and offer a wide range of partially observable tasks to train agents across different embodiments. Finally, we provide handcrafted metrics for each task and ship our benchmark with state-of-the-art baseline algorithms to ease evaluation and enable fast benchmarking.
Animation and machine learning research have shown great advancements in the past decade, leading to robust and powerful methods for learning complex physically-based animations. However, learning can take hours or days, especially if no reference movement data is available. In this paper, we propose and evaluate a novel combination of techniques for accelerating the learning of stable locomotion movements through self-imitation learning of synthetic animations. First, we produce synthetic and cyclic reference movement using a recent online tree search approach that can discover stable walking…
This paper considers learning robot locomotion and manipulation tasks from expert demonstrations. Generative adversarial imitation learning (GAIL) trains a discriminator that distinguishes expert from agent transitions, and in turn use a reward defined by the discriminator output to optimize a policy generator for the agent. This generative adversarial training approach is very powerful but depends on a delicate balance between the discriminator and the generator training. In high-dimensional problems, the discriminator training may easily overfit or exploit associations with task-irrelevant f…
This paper presents a new learning framework that leverages the knowledge from imitation learning, deep reinforcement learning, and control theories to achieve human-style locomotion that is natural, dynamic, and robust for humanoids. We proposed novel approaches to introduce human bias, i.e. motion capture data and a special Multi-Expert network structure. We used the Multi-Expert network structure to smoothly blend behavioral features, and used the augmented reward design for the task and imitation rewards. Our reward design is composable, tunable, and explainable by using fundamental concep…
Google Research（谷歌研究院）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Reproducing the diverse and agile locomotion skills of animals has been a longstanding challenge in robotics. While manually-designed controllers have been able to emulate many complex behaviors, building such controllers involves a time-consuming and difficult development process, often requiring substantial expertise of the nuances of each skill. Reinforcement learning provides an appealing alternative for automating the manual effort involved in the development of controllers. However, designing learning objectives that elicit the desired behaviors from an agent can also require a great dea…
Dynamic quadruped locomotion over challenging terrains with precise foot placements is a hard problem for both optimal control methods and Reinforcement Learning (RL). Non-linear solvers can produce coordinated constraint satisfying motions, but often take too long to converge for online application. RL methods can learn dynamic reactive controllers but require carefully tuned shaping rewards to produce good gaits and can have trouble discovering precise coordinated movements. Imitation learning circumvents this problem and has been used with motion capture data to extract quadruped gaits for …
Tsinghua University（清华大学）；Technical University of Munich（慕尼黑工业大学）
Not until recently, robust bipedal locomotion has been achieved through reinforcement learning. However, existing implementations rely heavily on insights and efforts from human experts, which is costly for the iterative design of robot systems. Also, styles of the learned motion are strictly limited to that of the reference. In this paper, we propose a new way to learn bipedal locomotion from a simple sine wave as the reference for foot heights. With the naive human insight that the two feet should be lifted up alternatively and periodically, we experimentally demonstrate on the Cassie robot …
In this work, we introduce a new method for imitation learning from video demonstrations. Our method, Relational Mimic (RM), improves on previous visual imitation learning methods by combining generative adversarial networks and relational learning. RM is flexible and can be used in conjunction with other recent advances in generative adversarial imitation learning to better address the need for more robust and sample-efficient approaches. In addition, we introduce a new neural network architecture that improves upon the previous state-of-the-art in reinforcement learning and illustrate how in…
We present an algorithm for Inverse Reinforcement Learning (IRL) from expert state observations only. Our approach decouples reward modelling from policy learning, unlike state-of-the-art adversarial methods which require updating the reward model during policy search and are known to be unstable and difficult to optimize. Our method, IL-flOw, recovers the expert policy by modelling state-state transitions, by generating rewards using deep density estimators trained on the demonstration trajectories, avoiding the instability issues of adversarial methods. We demonstrate that using the state tr…
University of California, Berkeley（加州大学伯克利分校）；Toyota Research Institute（丰田研究院）
On-policy imitation learning algorithms such as DAgger evolve a robot control policy by executing it, measuring performance (loss), obtaining corrective feedback from a supervisor, and generating the next policy. As the loss between iterations can vary unpredictably, a fundamental question is under what conditions this process will eventually achieve a converged policy. If one assumes the underlying trajectory distribution is static (stationary), it is possible to prove convergence for DAgger. However, in more realistic models for robotics, the underlying trajectory distribution is dynamic bec…
In recent years, a myriad of advanced results have been reported in the community of imitation learning, ranging from parametric to non-parametric, probabilistic to non-probabilistic and Bayesian to frequentist approaches. Meanwhile, ample applications (e.g., grasping tasks and human-robot collaborations) further show the applicability of imitation learning in a wide range of domains. While numerous literature is dedicated to the learning of human skills in unconstrained environment, the problem of learning constrained motor skills, however, has not received equal attention yet. In fact, const…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Vanderbilt University（范德堡大学）
We tackle the problem of perceptive locomotion in dynamic environments. In this problem, a quadrupedal robot must exhibit robust and agile walking behaviors in response to environmental clutter and moving obstacles. We present a hierarchical learning framework, named PRELUDE, which decomposes the problem of perceptive locomotion into high-level decision-making to predict navigation commands and low-level gait generation to realize the target commands. In this framework, we train the high-level navigation controller with imitation learning on human demonstrations collected on a steerable cart a…
The Hong Kong University of Science and Technology（香港科技大学）
Traditional model-based reinforcement learning (RL) methods generate forward rollout traces using the learnt dynamics model to reduce interactions with the real environment. The recent model-based RL method considers the way to learn a backward model that specifies the conditional probability of the previous state given the previous action and the current state to additionally generate backward rollout trajectories. However, in this type of model-based method, the samples derived from backward rollouts and those from forward rollouts are simply aggregated together to optimize the policy via th…
Guiding Energy-Efficient Locomotion through Impact Mitigation Rewards
Department of Electrical and Computer Engineering, Northeastern University（电气与计算机工程系，东北大学）；Department of Aerospace Engineering, California Institute of Technology（航空航天工程系，加州理工学院）
The aim in imitation learning is to learn effective policies by utilizing near-optimal expert demonstrations. However, high-quality demonstrations from human experts can be expensive to obtain in large numbers. On the other hand, it is often much easier to obtain large quantities of suboptimal or task-agnostic trajectories, which are not useful for direct imitation, but can nevertheless provide insight into the dynamical structure of the environment, showing what could be done in the environment even if not what should be done. We ask the question, is it possible to utilize such suboptimal off…
Safety-Aware Imitation Learning via MPC-Guided Disturbance Injection
Department of Electrical Engineering, Tsinghua University（电子工程系，清华大学）；Department of Electrical and Computer Engineering, University of Southern California（电气与计算机工程系，南加州大学）；Department of Aeronautics and Astronautics, Stanford University（航空与航天工程系，斯坦福大学）
Stanford University（斯坦福大学）；University of Washington（华盛顿大学）；Facebook AI Research（脸书人工智能研究院）
Reward function specification, which requires considerable human effort and iteration, remains a major impediment for learning behaviors through deep reinforcement learning. In contrast, providing visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions. This setting presents a number of challenges including repres…
Biomechanical Comparisons Reveal Divergence of Human and Humanoid Gaits
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences（中国科学院宁波材料技术与工程研究所）
Benchmarking the Full-Order Model Optimization Based Imitation in the Humanoid Robot Reinforcement Learning Walk
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Federal Research Center “Computer Science and Control” of Russian Academy of Sciences（俄罗斯科学院计算机科学与控制联邦研究中心）
Restoring Noisy Demonstration for Imitation Learning With Diffusion Models
Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）；Data Science Degree Program, National Taiwan University and Academia Sinica（国立台湾大学数据科学学士学位计划）；Department of Electrical Engineering, National Taiwan University（国立台湾大学电子工程系）
Fusion Dynamical Systems with Machine Learning in Imitation Learning: A Comprehensive Overview
Multi-Scale Medical Robotics Centre, Ltd.（多尺度医疗机器人中心有限公司）；The Chinese University of Hong Kong（香港中文大学）；Technical University of Munich（慕尼黑工业大学）；Mondragon Unibertsitatea（蒙德拉贡大学）；Nantong University（南通大学）
TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality
CNRS-AIST JRL (Joint Robotics Laboratory), IRL and National Institute of Advanced Industrial Science and Technology (AIST)（CNRS-AIST联合机器人实验室、IRL和国家先进工业科学与技术研究院）；Tokyo University of Science（东京科学大学）；University of Tsukuba（茨口大学）
Integrating Learning-Based Manipulation and Physics-Based Locomotion for Whole-Body Badminton Robot Control
School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；DeepCode Robotics（DeepCode机器人公司）；Department of Sports, Shanghai Jiao Tong University（上海交通大学体育学院）
We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. When the discriminator focuses on task-irrelevant features, it does not provide an informative reward signal, leading to poor task performance. We analyze this problem in detail and propose a solution that outperforms standard Generative Adversarial Imitation Learning (GAIL). Our proposed method, Task-Relevant Adversarial Imitation Learning (TRAIL), uses constrained discriminator optimization to learn informative rewa…
Northeastern University（东北大学）；Faculty of Robot Science and Engineering, Northeastern University（东北大学机器人科学与工程学院）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Foshan Graduate School, Northeastern University（东北大学佛山研究生院）
Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
Ecole Centrale de Lyon, CNRS, Universite Claude Bernard Lyon 1, INSA Lyon, Université Lumière Lyon 2, LIRIS, UMR5205（Lyon中央理工学院、CNRS、Claude Bernard Lyon 1大学、INSA Lyon、Lyon大学、LIRIS、UMR5205）；Beihang University（北航大学）；Dalian University of Technology（大连理工大学）；Institut Universitaire de France (IUF)（法国大学研究院（IUF））
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Harvard（哈佛大学）；CMU（卡内基梅隆大学）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
While significant research progress has been made in robot learning for control, unique challenges arise when simultaneously co-optimizing morphology. Existing work has typically been tailored for particular environments or representations. In order to more fully understand inherent design and performance tradeoffs and accelerate the development of new breeds of soft robots, a comprehensive virtual platform with well-established tasks, environments, and evaluation metrics is needed. In this work, we introduce SoftZoo, a soft robot co-design platform for locomotion in diverse environments. Soft…
IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning
Department of Computer Science（计算机科学系）；University of California Davis（加州大学戴维斯分校）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Toyota InfoTech Labs（丰田信息科技实验室）；Department of Mechanical and Aerospace Engineering（机械与航空航天工程系）
本文提出 IN-RIL，通过周期性交错 RL 与 IL 更新并用正交梯度分离避免冲突，显著提升机器人策略在线微调的稳定性、样本效率和成功率。
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
Institute for Intelligent Systems Research and Innovation (IISRI)（智能系统研究与创新研究所）；Deakin University（迪肯大学）；Harvard Paulson School of Engineering and Applied Sciences（哈佛保尔森工程与应用科学学院）；Harvard University（哈佛大学）
