arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.14288cs.ROcs.LGcs.SYeess.SY
收藏

具有稳定性保证的行动者-批评强化学习控制

Actor-Critic Reinforcement Learning for Control with Stability Guarantee

Harbin Institute of Technology(哈尔滨工业大学);University College London(伦敦大学学院);Delft University of Technology(代尔夫特理工大学)

Minghao Han, Lixian Zhang, Jun Wang, Wei Pan

摘要:

强化学习(RL)及其与深度学习的集成在各种机器人控制任务中取得了令人印象深刻的性能,从运动规划和导航到端到端的视觉操纵。然而,在无模型强化学习中,仅仅使用数据并不能保证稳定性。从控制理论的角度来看,稳定性是任何控制系统最重要的属性,因为它与机器人系统的安全性,鲁棒性和可靠性密切相关。在本文中,我们提出了一个行动者-评论家RL框架的控制,可以保证闭环稳定性,采用经典的李雅普诺夫方法在控制理论。首先,针对马尔可夫决策过程建模的随机非线性系统,提出了一个基于数据的稳定性定理。然后,我们表明,稳定性条件可以被利用作为行动者-评论家RL中的评论家来学习控制器/策略。最后,我们的方法的有效性进行了评估几个著名的三维机器人控制任务和合成生物学基因网络跟踪任务在三个不同的流行的物理仿真平台。作为对稳定性优势的实证评估,我们证明了学习策略可以使系统在受到系统参数变化和外部干扰等不确定性干扰时在一定程度上恢复到平衡点或路径点。

英文摘要:

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑