arXiv 2103.094072021-03-17eess.SYcs.SY

随机LQR控制器的无模型强化学习和原对偶优化设计

Model-Free Design of Stochastic LQR Controller from Reinforcement Learning and Primal-Dual Optimization Perspective

Man Li, Jiahu Qin, Wei Xing Zheng, Yaonan Wang, Yu Kang

摘要：

为了更好地理解各种强化学习（RL）算法的内在机制，并更好地利用优化理论推动RL的进一步发展，许多研究者开始重新审视线性二次型调节器（LQR）问题，其设置简单，但却捕捉到了RL的特点。受此启发，本文从RL和原-对偶优化的角度出发，研究了高斯噪声下线性系统的随机LQR控制器的无模型设计问题。从强化学习的角度出发，我们首先提出了一种新的无模型的策略外迭代算法（MF-OPPI），该算法通过重复使用采样数据来更新策略，从而在一定程度上缓解了数据饥渴问题。然后，我们提供了一个严格的分析算法的收敛性表明，所涉及的迭代是等价的，在经典的策略迭代（PI）算法的迭代。从优化的角度，我们首先将随机LQR问题转化为一个约束非凸优化问题，并证明了它具有很强的对偶性。然后，为了解决这个非凸优化问题，我们提出了一个基于模型的原始-对偶（MB-PD）算法的基础上产生的Karush-Kuhn-Tucker（KKT）条件的性质。我们还通过求解变换后的对偶可行性条件，给出了MB-PD算法的无模型实现。更重要的是，我们表明，在MB-PD算法的双重和原始更新步骤可以被解释为PI算法的政策评估和政策改进的步骤，分别。最后，我们提供了一个仿真例子来展示所提出的算法的性能。

英文摘要：

To further understand the underlying mechanism of various reinforcement learning (RL) algorithms and also to better use the optimization theory to make further progress in RL, many researchers begin to revisit the linear-quadratic regulator (LQR) problem, whose setting is simple and yet captures the characteristics of RL. Inspired by this, this work is concerned with the model-free design of stochastic LQR controller for linear systems subject to Gaussian noises, from the perspective of both RL and primal-dual optimization. From the RL perspective, we first develop a new model-free off-policy policy iteration (MF-OPPI) algorithm, in which the sampled data is repeatedly used for updating the policy to alleviate the data-hungry problem to some extent. We then provide a rigorous analysis for algorithm convergence by showing that the involved iterations are equivalent to the iterations in the classical policy iteration (PI) algorithm. From the perspective of optimization, we first reformulate the stochastic LQR problem at hand as a constrained non-convex optimization problem, which is shown to have strong duality. Then, to solve this non-convex optimization problem, we propose a model-based primal-dual (MB-PD) algorithm based on the properties of the resulting Karush-Kuhn-Tucker (KKT) conditions. We also give a model-free implementation for the MB-PD algorithm by solving a transformed dual feasibility condition. More importantly, we show that the dual and primal update steps in the MB-PD algorithm can be interpreted as the policy evaluation and policy improvement steps in the PI algorithm, respectively. Finally, we provide one simulation example to show the performance of the proposed algorithms.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文