A Differential Dynamic Programming Framework for Inverse Reinforcement Learning
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School for Engineering of Matter, Transport, and Energy, Arizona State University（亚利桑那州立大学物质、运输与能源工程学院）；Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（KTH皇家理工学院电气工程与计算机科学学院决策与控制系统分部）；Digital Futures（数字未来（机构名））
Kun Cao, Xinhang Xu, Wanxin Jin, Karl H. Johansson, Lihua Xie
备注：
20 pages, 15 figures; submitted to IEEE for potential publication
A differential dynamic programming (DDP)-based framework for inverse reinforcement learning (IRL) is introduced to recover the parameters in the cost function, system dynamics, and constraints from demonstrations. Different from existing work, where DDP was used for the inner forward problem with inequality constraints, our proposed framework uses it for efficient computation of the gradient required in the outer inverse problem with equality and inequality constraints. The equivalence between the proposed method and existing methods based on Pontryagin's Maximum Principle (PMP) is established. More importantly, using this DDP-based IRL with an open-loop loss function, a closed-loop IRL framework is presented. In this framework, a loss function is proposed to capture the closed-loop nature of demonstrations. It is shown to be better than the commonly used open-loop loss function. We show that the closed-loop IRL framework reduces to a constrained inverse optimal control problem under certain assumptions. Under these assumptions and a rank condition, it is proven that the learning parameters can be recovered from the demonstration data. The proposed framework is extensively evaluated through four numerical robot examples and one real-world quadrotor system. The experiments validate the theoretical results and illustrate the practical relevance of the approach.
