arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02072eess.SYcs.SY
收藏

面向风险敏感线性二次高斯控制的鲁棒强化学习

Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control

Leilei Cui, Tamer Başar, Zhong-Ping Jiang

AI总结:

针对模型失配的离散时间线性系统,提出基于风险敏感LQG控制的双环鲁棒策略优化算法,并给出未知动力学下的无模型离策略版本,证明其收敛性与小扰动ISS。

摘要:

本文针对可能由仿真到现实差距导致模型失配的离散时间线性系统,提出了一种新颖的鲁棒强化学习框架。关键策略是运用控制理论中的先进技术。借助经典风险敏感线性二次高斯(LQG)控制的表述,提出了一种双环策略优化算法,以生成鲁棒最优控制器。该双环策略优化算法被证明具有全局一致收敛性,并且在学习过程中对扰动具有鲁棒性。这一鲁棒性被称为小扰动输入到状态稳定性,保证只要每个学习步骤的扰动相对较小,所提出的策略优化算法就会收敛到最优控制器的一个小邻域内。此外,当系统动力学未知时,提出了一种新颖的无模型离策略策略优化算法。最后,通过数值算例对所提出的算法进行了说明。

英文摘要:

This paper proposes a novel robust reinforcement learning framework for discrete-time linear systems with model mismatch that may arise from the sim-to-real gap. A key strategy is to invoke advanced techniques from control theory. Using the formulation of the classical risk-sensitive linear quadratic Gaussian control, a dual-loop policy optimization algorithm is proposed to generate a robust optimal controller. The dual-loop policy optimization algorithm is shown to be globally and uniformly convergent, and robust against disturbances during the learning process. This robustness property is called small-disturbance input-to-state stability and guarantees that the proposed policy optimization algorithm converges to a small neighborhood of the optimal controller as long as the disturbance at each learning step is relatively small. In addition, when the system dynamics is unknown, a novel model-free off-policy policy optimization algorithm is proposed. Finally, numerical examples are provided to illustrate the proposed algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑