Continuous Control of Diverse Skills in Quadruped Robots Without Complete Expert Datasets
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Beijing Jingcheng Zhitong Robotics Technology Co.（北京京城智通机器人科技有限公司）；Engineering Research Center of AI and Robotics（AI与机器人工程研究中心）
Learning diverse skills for quadruped robots presents significant challenges, such as mastering complex transitions between different skills and handling tasks of varying difficulty. Existing imitation learning methods, while successful, rely on expensive datasets to reproduce expert behaviors. Inspired by introspective learning, we propose Progressive Adversarial Self-Imitation Skill Transition (PASIST), a novel method that eliminates the need for complete expert datasets. PASIST autonomously explores and selects high-quality trajectories based on predefined target poses instead of demonstrations, leveraging the Generative Adversarial Self-Imitation Learning (GASIL) framework. To further enhance learning, We develop a skill selection module to mitigate mode collapse by balancing the weights of skills with varying levels of difficulty. Through these methods, PASIST is able to reproduce skills corresponding to the target pose while achieving smooth and natural transitions between them. Evaluations on both simulation platforms and the Solo 8 robot confirm the effectiveness of PASIST, offering an efficient alternative to expert-driven learning.
Diverse Skill Discovery for Quadruped Robots via Unsupervised Learning
College of Intelligent Robotics and Advanced Manufacturing（智能机器人与先进制造学院）；Fudan University（复旦大学）；College of Foreign Language（外国语言学院）；The University of Shanghai for Science and Technology（上海科技大学）
Seoul National University（首尔国立大学）；Georgia Institute of Technology（佐治亚理工学院）
A motion-based control interface promises flexible robot operations in dangerous environments by combining user intuitions with the robot's motor capabilities. However, designing a motion interface for non-humanoid robots, such as quadrupeds or hexapods, is not straightforward because different dynamics and control strategies govern their movements. We propose a novel motion control system that allows a human user to operate various motor tasks seamlessly on a quadrupedal robot. We first retarget the captured human motion into the corresponding robot motion with proper semantics using supervis…
In recent years, reinforcement learning (RL) has shown outstanding performance for locomotion control of highly articulated robotic systems. Such approaches typically involve tedious reward function tuning to achieve the desired motion style. Imitation learning approaches such as adversarial motion priors aim to reduce this problem by encouraging a pre-defined motion style. In this work, we present an approach to augment the concept of adversarial motion prior-based RL to allow for multiple, discretely switchable styles. We show that multiple styles and skills can be learned simultaneously wit…
We present a learning algorithm for training a single policy that imitates multiple gaits of a walking robot. To achieve this, we use and extend MPC-Net, which is an Imitation Learning approach guided by Model Predictive Control (MPC). The strategy of MPC-Net differs from many other approaches since its objective is to minimize the control Hamiltonian, which derives from the principle of optimality. To represent the policies, we employ a mixture-of-experts network (MEN) and observe that the performance of a policy improves if each expert of a MEN specializes in controlling exactly one mode of …
Georgia Institute of Technology（佐治亚理工学院）；Google（谷歌）；Stanford University（斯坦福大学）
Outdoor navigation on sidewalks in urban environments is the key technology behind important human assistive applications, such as last-mile delivery or neighborhood patrol. This paper aims to develop a quadruped robot that follows a route plan generated by public map services, while remaining on sidewalks and avoiding collisions with obstacles and pedestrians. We devise a two-staged learning framework, which first trains a teacher agent in an abstract world with privileged ground-truth information, and then applies Behavior Cloning to teach the skills to a student agent who only has access to…
Spatio-Temporal Motion Retargeting for Quadruped Robots
Department of Artificial Intelligence, Korea University（人工智能系，韩国大学）；Department of Computer Science, ETH Zurich（计算机科学系，苏黎世联邦理工学院）；Department of Mechanical and Aerospace Engineering, UCLA（机械与航空航天工程系，加州大学洛杉矶分校）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
Learning diverse skills is one of the main challenges in robotics. To this end, imitation learning approaches have achieved impressive results. These methods require explicitly labeled datasets or assume consistent skill execution to enable learning and active control of individual behaviors, which limits their applicability. In this work, we propose a cooperative adversarial method for obtaining single versatile policies with controllable skill sets from unlabeled datasets containing diverse state transition patterns by maximizing their discriminability. Moreover, we show that by utilizing un…
LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning
Department of Translational Surgical Oncology, NCT/UCC Dresden（转化外科肿瘤学系）；Cluster of Excellence-CeTI, TUD, Germany（卓越中心-CeTI）；Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany（visceral、胸腔和血管外科系）
Google Research（谷歌研究院）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Reproducing the diverse and agile locomotion skills of animals has been a longstanding challenge in robotics. While manually-designed controllers have been able to emulate many complex behaviors, building such controllers involves a time-consuming and difficult development process, often requiring substantial expertise of the nuances of each skill. Reinforcement learning provides an appealing alternative for automating the manual effort involved in the development of controllers. However, designing learning objectives that elicit the desired behaviors from an agent can also require a great dea…
Motion Priors Reimagined: Adapting Flat-Terrain Skills for Complex Quadruped Mobility
Department of Mechanical Engineering, EPFL（瑞士联邦理工学院机械工程系）；ETH AI Center, ETH Zurich（苏黎世联邦理工学院人工智能中心）；Robotic Systems Lab, ETH Zurich（苏黎世联邦理工学院机器人系统实验室）
The University of Tokyo（东京大学）；ETH Zürich（苏黎世联邦理工学院）
Quadrupedal robots are skillful at locomotion tasks while lacking manipulation skills, not to mention dexterous manipulation abilities. Inspired by the animal behavior and the duality between multi-legged locomotion and multi-fingered manipulation, we showcase a circus ball challenge on a quadrupedal robot, ANYmal. We employ a model-free reinforcement learning approach to train a deep policy that enables the robot to balance and manipulate a light-weight ball robustly using its limbs without any contact measurement sensor. The policy is trained in the simulation, in which we randomize many phy…
Georgia Institute of Technology（佐治亚理工学院）；Meta Platforms, Inc.（Meta平台公司）
Learning various motor skills for quadrupedal robots is a challenging problem that requires careful design of task-specific mathematical models or reward descriptions. In this work, we propose to learn a single capable policy using deep reinforcement learning by imitating a large number of reference motions, including walking, turning, pacing, jumping, sitting, and lying. On top of the existing motion imitation framework, we first carefully design the observation space, the action space, and the reward function to improve the scalability of the learning as well as the robustness of the final p…
University of California, Berkeley（加州大学伯克利分校）；Université de Montréal（蒙特利尔大学）；Mila（米拉人工智能研究院）
We address the problem of enabling quadrupedal robots to perform precise shooting skills in the real world using reinforcement learning. Developing algorithms to enable a legged robot to shoot a soccer ball to a given target is a challenging problem that combines robot motion control and planning into one task. To solve this problem, we need to consider the dynamics limitation and motion stability during the control of a dynamic legged robot. Moreover, we need to consider motion planning to shoot the hard-to-model deformable ball rolling on the ground with uncertain friction to a desired locat…
Robots operating in human environments need various skills, like slow and fast walking, turning, side-stepping, and many more. However, building robot controllers that can exhibit such a large range of behaviors is a challenging problem that requires tedious investigation for every task. We present a unified model-based control algorithm for imitating different animal gaits without expensive simulation training or real-world fine-tuning. Our method consists of stance and swing leg controllers using a centroidal dynamics model augmented with online adaptation techniques. We also develop a whole…
There is a dramatic shortage of skilled labor for modern vineyards. The Vinum project is developing a mobile robotic solution to autonomously navigate through vineyards for winter grapevine pruning. This necessitates an autonomous navigation stack for the robot pruning a vineyard. The Vinum project is using the quadruped robot HyQReal. This paper introduces an architecture for a quadruped robot to autonomously move through a vineyard by identifying and approaching grapevines for pruning. The higher level control is a state machine switching between searching for destination positions, autonomo…
Robotics at Google（谷歌机器人）；Georgia Institute of Technology（佐治亚理工学院）
Developing agile behaviors for legged robots remains a challenging problem. While deep reinforcement learning is a promising approach, learning truly agile behaviors typically requires tedious reward shaping and careful curriculum design. We formulate agile locomotion as a multi-stage learning problem in which a mentor guides the agent throughout the training. The mentor is optimized to place a checkpoint to guide the movement of the robot's center of mass while the student (i.e. the robot) learns to reach these checkpoints. Once the student can solve the task, we teach the student to perform …
University of California, Berkeley（加州大学伯克利分校）；Simon Fraser University（西蒙弗雷泽大学）；Georgia Institute of Technology（佐治亚理工学院）
We present a reinforcement learning (RL) framework that enables quadrupedal robots to perform soccer goalkeeping tasks in the real world. Soccer goalkeeping using quadrupeds is a challenging problem, that combines highly dynamic locomotion with precise and fast non-prehensile object (ball) manipulation. The robot needs to react to and intercept a potentially flying ball using dynamic locomotion maneuvers in a very short amount of time, usually less than one second. In this paper, we propose to address this problem using a hierarchical model-free RL framework. The first component of the framewo…
Tsinghua University（清华大学）；University of Edinburgh（爱丁堡大学）
For model-free deep reinforcement learning of quadruped locomotion, the initialization of robot configurations is crucial for data efficiency and robustness. This work focuses on algorithmic improvements of data efficiency and robustness simultaneously through automatic discovery of initial states, which is achieved by our proposed K-Access algorithm based on accessibility metrics. Specifically, we formulated accessibility metrics to measure the difficulty of transitions between two arbitrary states, and proposed a novel K-Access algorithm for state-space clustering that automatically discover…
University of Notre Dame（圣母大学）；Robotics at Google（谷歌机器人部门）
Specialized motions such as jumping are often achieved on quadruped robots by solving a trajectory optimization problem once and executing the trajectory using a tracking controller. This approach is in parallel with Model Predictive Control (MPC) strategies that commonly control regular gaits via online re-planning. In this work, we present a nonlinear MPC (NMPC) technique that unlocks on-the-fly re-planning of specialized motion skills and regular locomotion within a unified framework. The NMPC reasons about a hybrid kinodynamic model, and is solved using a variant of a constrained Different…
Latent Conditioned Loco-Manipulation Using Motion Priors
LAAS-CNRS, Université de Toulouse（CNRS拉法耶特研究所、图卢兹大学）；Artificial and Natural Intelligence Toulouse Institute（图卢兹人工智能与自然智能研究所）；University of Edinburgh, School of Informatics（爱丁堡大学信息学院）
End-to-End Multi-Task Policy Learning from NMPC for Quadruped Locomotion
Institute for Artificial Intelligence and Autonomous Systems (A2S)（人工智能与自主系统研究所）；Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用科学大学）；Humanoid Robots Lab（人形机器人实验室）；University of Bonn（波恩大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（Lamarr机器学习与人工智能研究所）；Center for Robotics（机器人中心）；Fraunhofer Institute for Intelligent Analysis and Information Systems（弗劳恩霍夫智能分析与信息系统研究所）
Georgia Institute of Technology（佐治亚理工学院）；Facebook AI Research（Facebook人工智能研究院）
Recent work has shown results on learning navigation policies for idealized cylinder agents in simulation and transferring them to real wheeled robots. Deploying such navigation policies on legged robots can be challenging due to their complex dynamics, and the large dynamical difference between cylinder agents and legged systems. In this work, we learn hierarchical navigation policies that account for the low-level dynamics of legged robots, such as maximum speed, slipping, contacts, and learn to successfully navigate cluttered indoor environments. To enable transfer of policies learned in si…
Indian Institute of Technology-Madras（印度理工学院马德拉斯分校）；Indian Institute of Science（印度科学理工学院）
With the research into development of quadruped robots picking up pace, learning based techniques are being explored for developing locomotion controllers for such robots. A key problem is to generate leg trajectories for continuously varying target linear and angular velocities, in a stable manner. In this paper, we propose a two pronged approach to address this problem. First, multiple simpler policies are trained to generate trajectories for a discrete set of target velocities and turning radius. These policies are then augmented using a higher level neural network for handling the transiti…
Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Smart Systems Institute, NUS（NUS智能系统研究所）；Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Beijing Zhongguancun Academy（北京中关村学院）
University of Notre Dame（圣母大学）；Robotics at Google（谷歌机器人部门）
Reinforcement Learning (RL) has witnessed great strides for quadruped locomotion, with continued progress in the reliable sim-to-real transfer of policies. However, it remains a challenge to reuse a policy on another robot, which could save time for retraining. In this work, we present a framework for zero-shot policy retargeting wherein diverse motor skills can be transferred between robots of different shapes and sizes. The new framework centers on a planning-and-control pipeline that systematically integrates RL and Model Predictive Control (MPC). The planning stage employs RL to generate a…
