Real-time synthesis of legged locomotion maneuvers in challenging industrial settings is still an open problem, requiring simultaneous determination of footsteps locations several steps ahead while generating whole-body motions close to the robot's limits. State estimation and perception errors impose the practical constraint of fast re-planning motions in a model predictive control (MPC) framework. We first observe that the computational limitation of perceptive locomotion pipelines lies in the combinatorics of contact surface selection. Re-planning contact locations on selected surfaces can be accomplished at MPC frequencies (50-100 Hz). Then, whole-body motion generation typically follows a reference trajectory for the robot base to facilitate convergence. We propose removing this constraint to robustly address unforeseen events such as contact slipping, by leveraging a state-of-the-art whole-body MPC (Croccodyl). Our contributions are integrated into a complete framework for perceptive locomotion, validated under diverse terrain conditions, and demonstrated in challenging trials that push the robot's actuation limits, as well as in the ICRA 2023 quadruped challenge simulation.
As both legged robots and embedded compute have become more capable, researchers have started to focus on field deployment of these robots. Robust autonomy in unstructured environments requires perception of the world around the robot in order to avoid hazards. However, incorporating perception online while maintaining agile motion is more challenging for legged robots than other mobile robots due to the complex planners and controllers required to handle the dynamics of locomotion. This report will compare three recent approaches for perceptive locomotion and discuss the different ways in whi…
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；University College London（伦敦大学学院）
This work developed a learning framework for perceptive legged locomotion that combines visual feedback, proprioceptive information, and active gait regulation of foot-ground contacts. The perception requires only one forward-facing camera to obtain the heightmap, and the active regulation of gait paces and traveling velocity are realized through our formulation of CPG-based high-level imitation of foot-ground contacts. Through this framework, an end-user has the ability to command task-level inputs to control different walking speeds and gait frequencies according to the traversal of differen…
ETH Zurich（苏黎世联邦理工学院）；École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Dynamic locomotion in rough terrain requires accurate foot placement, collision avoidance, and planning of the underactuated dynamics of the system. Reliably optimizing for such motions and interactions in the presence of imperfect and often incomplete perceptive information is challenging. We present a complete perception, planning, and control pipeline, that can optimize motions for all degrees of freedom of the robot in real-time. To mitigate the numerical challenges posed by the terrain a sequence of convex inequality constraints is extracted as local approximations of foothold feasibility…
Oxford Robotics Institute, University of Oxford（牛津大学牛津机器人研究所）
To dynamically traverse challenging terrain, legged robots need to continually perceive and reason about upcoming features, adjust the locations and timings of future footfalls and leverage momentum strategically. We present a pipeline that enables flexibly-parametrized trajectories for perceptive and dynamic quadruped locomotion to be optimized in an online, receding-horizon manner. The initial guess passed to the optimizer affects the computation needed to achieve convergence and the quality of the solution. We consider two methods for generating good guesses. The first is a heuristic initia…
PolygMap: A Perceptive Locomotion Framework for Humanoid Robot Stair Climbing
The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人社会研究院）；Electronic science and technology, Guangdong University of Technology（广东技术大学电子科学与技术学院）；Leju robotics（乐居机器人）
Robotic Systems Lab, ETH Zurich（苏黎世联邦理工学院机器人系统实验室）；Robotics and Artificial Intelligence Lab, KAIST（韩国科学技术院机器人与人工智能实验室）；Intelligent Systems Lab, Intel（英特尔智能系统实验室）
Legged robots that can operate autonomously in remote and hazardous environments will greatly increase opportunities for exploration into under-explored areas. Exteroceptive perception is crucial for fast and energy-efficient locomotion: perceiving the terrain before making contact with it enables planning and adaptation of the gait ahead of time to maintain speed and stability. However, utilizing exteroceptive perception robustly for locomotion has remained a grand challenge in robotics. Snow, vegetation, and water visually appear as obstacles on which the robot cannot step~-- or are missing …
Gait-Adaptive Perceptive Humanoid Locomotion with Real-Time Under-Base Terrain Reconstruction
Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China（电子工程与信息科学系，中国科学技术大学）；LimX Dynamics（LimX动力学）；Hong Kong University of Science and Technology（香港科技大学）；School of Mechanics Engineering, Harbin Institute of Technology (HIT)（机械工程学院，哈尔滨工业大学）；Department of Computing, The Hong Kong Polytechnic University（计算学院，香港理工大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute (ZJUI)（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）
PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes
School of Automation and Intelligent Sensing（自动化与智能感知学院）；Shanghai Jiao Tong University（上海交通大学）；Global College（全球学院）；school of Computer Science（计算机科学学院）；Shanghai Key Laboratory of Navigation and Location Based Services（导航与位置基于服务上海市重点实验室）
Laboratoire d’Informatique, de Robotique et de Microélectronique de Montpellier (LIRMM), CNRS–University of Montpellier（蒙彼利埃计算机科学、机器人技术与微电子实验室（LIRMM），法国国家科学研究中心–蒙彼利埃大学）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）
We present a model predictive controller (MPC) for multi-contact locomotion where predictive optimizations are realized by time-optimal path parameterization (TOPP). A key feature of this solution is that, contrary to existing planners where step timings are provided as inputs, here the timing between contact switches is computed as output of a fast nonlinear optimization. This is particularly appealing to multi-contact locomotion, where proper timings depend on terrain topology and suitable heuristics are unknown. We show how to formulate legged locomotion as a TOPP problem and demonstrate th…
PGTT: Phase-Guided Terrain Traversal for Perceptive Legged Locomotion
Laboratory of Automation and Robotics (LAR) in the Department of Electrical & Computer Engineering, University of Patras（自动化与机器人实验室（LAR）（电气与计算机工程系，帕特拉斯大学））；Archimedes/Athena RC, Greece（阿基米德/雅典娜RC，希腊）；Athena - Research and Innovation Center in Information, Communication and Knowledge Technologies, Xanthi, Greece（雅典娜信息、通信和知识技术研究中心（Xanthi，希腊））；Computational Intelligence Laboratory (CILab), Department of Mathematics, University of Patras（计算智能实验室（CILab）（数学系，帕特拉斯大学））
In this paper, we present a real-time whole-body planner for collision-free legged mobile manipulation. We enforce both self-collision and environment-collision avoidance as soft constraints within a Model Predictive Control (MPC) scheme that solves a multi-contact optimal control problem. By penalizing the signed distances among a set of representative primitive collision bodies, the robot is able to safely execute a variety of dynamic maneuvers while preventing any self-collisions. Moreover, collision-free navigation and manipulation in both static and dynamic environments are made viable th…
University of Edinburgh（爱丁堡大学）；University College London（伦敦大学学院）
This work developed a kernel-based residual learning framework for quadrupedal robotic locomotion. Initially, a kernel neural network is trained with data collected from an MPC controller. Alongside a frozen kernel network, a residual controller network is trained via reinforcement learning to acquire generalized locomotion skills and resilience against external perturbations. With this proposed framework, a robust quadrupedal locomotion controller is learned with high sample efficiency and controllability, providing omnidirectional locomotion at continuous velocities. Its versatility and robu…
Istituto Italiano di Tecnologia（意大利理工学院）；University of Notre Dame（圣母大学）
We present a novel control strategy for dynamic legged locomotion in complex scenarios, that considers information about the morphology of the terrain in contexts when only on-board mapping and computation are available. The strategy is built on top of two main elements: first a contact sequence task that provides safe foothold locations based on a convolutional neural network to perform fast and continuous evaluation of the terrain in search of safe foothold locations; then a model predictive controller that considers the foothold locations given by the contact sequence task to optimize targe…
Dynamic legged locomotion is a challenging topic because of the lack of established control schemes which can handle aerial phases, short stance times, and high-speed leg swings. In this paper, we propose a controller combining whole-body control (WBC) and model predictive control (MPC). In our framework, MPC finds an optimal reaction force profile over a longer time horizon with a simple model, and WBC computes joint torque, position, and velocity commands based on the reaction forces computed from MPC. Unlike existing WBCs, which attempt to track commanded body trajectories, our controller i…
Istituto Italiano di Tecnologia（意大利理工学院）；University of Genoa（热那亚大学）；University of Manchester（曼彻斯特大学）
This paper presents a Non-Linear Model Predictive Controller for humanoid robot locomotion with online step adjustment capabilities. The proposed controller considers the Centroidal Dynamics of the system to compute the desired contact forces and torques and contact locations. Differently from bipedal walking architectures based on simplified models, the presented approach considers the reduced centroidal model, thus allowing the robot to perform highly dynamic movements while keeping the control problem still treatable online. We show that the proposed controller can automatically adjust the …
Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain
Mondo Robotics；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Artificial General Intelligence Institute, University of Science and Technology of China（中国科学技术大学通用人工智能研究院）
Beijing Research Institute of UBTECH Robotics（优必选机器人北京研究院）；Tsinghua University（清华大学）
The planning of whole-body motion and step time for bipedal locomotion is constructed as a model predictive control (MPC) problem, in which a sequence of optimization problems needs to be solved online. While directly solving these problems is extremely time-consuming, we propose a predictive gait synthesizer to offer immediate solutions. Based on the full-dimensional model, a library of gaits with different speeds and periods is first constructed offline. Then the proposed gait synthesizer generates real-time gaits at 1kHz by synthesizing the gait library based on the online prediction of cen…
Convex model predictive controls (MPCs) with a single rigid body model have demonstrated strong performance on real legged robots. However, convex MPCs are limited by their assumptions such as small rotation angle and pre-defined gait, limiting the richness of potential solutions. We remove those assumptions and solve the complete mixed-integer non-convex programming with single rigid body model. We first collect datasets of pre-solved problems offline, then learn the problem-solution map to solve this optimization fast for MPC. If warm-starts can be found, offline problems can be solved close…
ETH Zurich（苏黎世联邦理工学院）；Robotic Systems Lab（机器人系统实验室）
In this paper, we propose a whole-body planning framework that unifies dynamic locomotion and manipulation tasks by formulating a single multi-contact optimal control problem. We model the hybrid nature of a generic multi-limbed mobile manipulator as a switched system, and introduce a set of constraints that can encode any pre-defined gait sequence or manipulation schedule in the formulation. Since the system is designed to actively manipulate its environment, the equations of motion are composed by augmenting the robot's centroidal dynamics with the manipulated-object dynamics. This allows us…
Dynamic Modeling and MPC for Locomotion of Tendon-Driven Soft Quadruped
Department of Electrical Engineering, IIT Gandhinagar（电子工程系，印度加尔各直辖区理工学院）；Department of Electronics and Computer Engineering, Sreenidhi Institute of Science and Technology（电子与计算机工程系，斯里尼迪学院科学与技术研究所）；Department of Mechanical Engineering, IIT Gandhinagar（机械工程系，印度加尔各直辖区理工学院）
Virginia Tech（弗吉尼亚理工大学）；California Institute of Technology（加州理工学院）
This paper presents a layered control approach for real-time trajectory planning and control of robust cooperative locomotion by two holonomically constrained quadrupedal robots. A novel interconnected network of reduced-order models, based on the single rigid body (SRB) dynamics, is developed for trajectory planning purposes. At the higher level of the control architecture, two different model predictive control (MPC) algorithms are proposed to address the optimal control problem of the interconnected SRB dynamics: centralized and distributed MPCs. The distributed MPC assumes two local quadra…
In this paper, previous works on the Model Predictive Control (MPC) and the Divergent Component of Motion (DCM) for bipedal walking control are extended. To this end, we employ a single MPC which uses a combination of Center of Pressure (CoP) manipulation, step adjustment, and Centroidal Moment Pivot (CMP) modulation to design a robust walking controller. Furthermore, we exploit the concept of time-varying DCM to generalize our walking controller for walking in uneven surfaces. Using our scheme, a general and robust walking controller is designed which can be implemented on robots with differe…
LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots
School of Automation and Intelligent Manufacturing (AIM), Southern University of Science and Technology（自动化与智能制造学院，南方科技大学）；Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（土木与环境工程系，香港科技大学）；Department of Mechanical Engineering, The University of Hong Kong（机械工程系，香港大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute (ZJUI), Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所，浙江大学）；LimX Dynamics, Shenzhen, China（深圳LimX动力有限公司）
Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion
New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校）；Georgia Institute of Technology（佐治亚理工学院）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）
