arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.12115cs.LGcs.RO
收藏

基于蒙特卡罗梯度估计的模型策略搜索及其在实际系统中的应用

Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Application

University of Padova(帕多瓦大学);Mitsubishi Electric Research Laboratories(三菱电机研究实验室)

Fabio Amadio, Alberto Dalla Libera, Riccardo Antonello, Daniel Nikovski, Ruggero Carli, Diego Romeres

摘要:

在本文中,我们提出了一个基于模型的强化学习(MBRL)算法命名为\emph{Monte Carlo Probabilistic Inference for Learning COntrol}(MC-PILCO)。该算法依赖于高斯过程(GP)的系统动态模型和蒙特卡罗方法来估计的策略梯度。这定义了一个框架,在这个框架中,我们取消了以下组件的选择:(i)成本函数的选择,(ii)使用dropout优化策略,(iii)通过在GP模型中使用结构化内核来提高数据效率。上述方面的组合显著地影响MC-PILCO的性能。在一个模拟的车杆环境中的数值比较表明,MC-PILCO具有更好的数据效率和控制性能w.r.t.最先进的基于GP的MBRL算法最后,我们将MC-PILCO应用到实际系统中,特别是考虑部分可测状态的系统。我们讨论了在政策优化过程中的测量系统和状态估计建模的重要性。所提出的解决方案的有效性已在模拟和两个真实的系统,一个古田摆和球板钻机进行了测试。

英文摘要:

In this paper, we present a Model-Based Reinforcement Learning (MBRL) algorithm named \emph{Monte Carlo Probabilistic Inference for Learning COntrol} (MC-PILCO). The algorithm relies on Gaussian Processes (GPs) to model the system dynamics and on a Monte Carlo approach to estimate the policy gradient. This defines a framework in which we ablate the choice of the following components: (i) the selection of the cost function, (ii) the optimization of policies using dropout, (iii) an improved data efficiency through the use of structured kernels in the GP models. The combination of the aforementioned aspects affects dramatically the performance of MC-PILCO. Numerical comparisons in a simulated cart-pole environment show that MC-PILCO exhibits better data efficiency and control performance w.r.t. state-of-the-art GP-based MBRL algorithms. Finally, we apply MC-PILCO to real systems, considering in particular systems with partially measurable states. We discuss the importance of modeling both the measurement system and the state estimators during policy optimization. The effectiveness of the proposed solutions has been tested in simulation and on two real systems, a Furuta pendulum and a ball-and-plate rig.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑