arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1311.0396eess.SYcs.SYmath.OCstat.ML
收藏

基于数据的近似策略迭代法用于非线性连续时间最优控制设计

Data-based approximate policy iteration for nonlinear continuous-time optimal control design

Beihang University (Beijing University of Aeronautics and Astronautics)(北京航空航天大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)

Biao Luo, Huai-Ning Wu, Tingwen Huang, Derong Liu

AI总结:

本文提出基于数据的近似策略迭代方法,用于解决非线性最优控制问题,通过演员-评论家结构神经网络近似控制策略和成本函数,无需系统数学模型即可学习HJB方程解。

摘要:

本文针对无模型非线性最优问题,开发了一种基于数据的强化学习技术。已知非线性最优控制问题依赖于哈密顿-雅可比-贝尔曼(HJB)方程的解,该方程通常是无法解析求解的非线性偏微分方程。更糟糕的是,大多数实际系统过于复杂,无法建立准确的数学模型。为克服这些困难,本文提出了一种基于数据的近似策略迭代(API)方法,利用实际系统数据而非系统模型。首先,推导了一个用于约束最优控制问题的无模型策略迭代算法,并证明了其收敛性,该算法可以学习HJB方程和最优控制策略的解,而无需任何系统数学模型的知识。该算法的实现基于演员-评论家结构的思想,其中演员和评论家神经网络(NNs)分别用于近似控制策略和成本函数。为了更新演员和评论家NNs的权重,开发了一种基于残差加权方法的最小二乘方法。整个基于数据的API方法包括两部分,其中第一部分在线实施以收集实际系统信息,第二部分进行离线策略迭代以学习HJB方程和控制策略的解。然后,将该基于数据的API算法简化以解决非线性和线性系统的无约束最优控制问题。最后,在简单的非线性系统上测试了该基于数据的API控制设计方法的效率,并进一步应用于旋转/平移执行器系统。仿真结果证明了所提方法的有效性。

英文摘要:

This paper addresses the model-free nonlinear optimal problem with generalized cost functional, and a data-based reinforcement learning technique is developed. It is known that the nonlinear optimal control problem relies on the solution of the Hamilton-Jacobi-Bellman (HJB) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, most of practical systems are too complicated to establish their accurate mathematical model. To overcome these difficulties, we propose a data-based approximate policy iteration (API) method by using real system data rather than system model. Firstly, a model-free policy iteration algorithm is derived for constrained optimal control problem and its convergence is proved, which can learn the solution of HJB equation and optimal control policy without requiring any knowledge of system mathematical model. The implementation of the algorithm is based on the thought of actor-critic structure, where actor and critic neural networks (NNs) are employed to approximate the control policy and cost function, respectively. To update the weights of actor and critic NNs, a least-square approach is developed based on the method of weighted residuals. The whole data-based API method includes two parts, where the first part is implemented online to collect real system information, and the second part is conducting offline policy iteration to learn the solution of HJB equation and the control policy. Then, the data-based API algorithm is simplified for solving unconstrained optimal control problem of nonlinear and linear systems. Finally, we test the efficiency of the data-based API control design method on a simple nonlinear system, and further apply it to a rotational/translational actuator system. The simulation results demonstrate the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑