arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1903.09688cs.LGcs.NEcs.SYeess.SYstat.ML
收藏

强化学习的符号回归方法

Symbolic Regression Methods for Reinforcement Learning

Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague(布拉格捷克理工大学信息学、机器人与控制论研究所);Faculty of Electrical Engineering, Czech Technical University in Prague(布拉格捷克理工大学电气工程学院);Cognitive Robotics, Delft University of Technology(代尔夫特理工大学认知机器人系)

Jiří Kubalík, Erik Derner, Jan Žegklitz, Robert Babuška

摘要:

强化学习算法可以解决动态决策和最优控制问题。对于连续值的状态和输入变量,强化学习算法必须依赖于函数逼近器来表示价值函数和策略映射。常用的数值逼近器,如神经网络或基函数展开,有两个主要缺点:它们是黑箱模型,对所学习的映射几乎没有洞察力,并且它们需要对其超参数进行大量的试错调整。本文提出了一种利用符号回归构造解析表达式形式的光滑价值函数的新方法。我们介绍了三个离线的方法来寻找价值函数的基础上的状态转换模型:符号值迭代,符号政策迭代,和直接的Bellman方程的解决方案。四个非线性控制问题:摩擦下的速度控制,一个链接和两个链接摆摆动,和磁操纵的方法进行说明。结果表明,价值函数产生性能良好的政策,是紧凑的,数学上易于处理,并容易插入到其他算法。这使得它们潜在地适合于闭环系统的进一步分析。与另一种方法使用神经网络的比较表明,我们的方法优于神经网络为基础的。

英文摘要:

Reinforcement learning algorithms can solve dynamic decision-making and optimal control problems. With continuous-valued state and input variables, reinforcement learning algorithms must rely on function approximators to represent the value function and policy mappings. Commonly used numerical approximators, such as neural networks or basis function expansions, have two main drawbacks: they are black-box models offering little insight into the mappings learned, and they require extensive trial and error tuning of their hyper-parameters. In this paper, we propose a new approach to constructing smooth value functions in the form of analytic expressions by using symbolic regression. We introduce three off-line methods for finding value functions based on a state-transition model: symbolic value iteration, symbolic policy iteration, and a direct solution of the Bellman equation. The methods are illustrated on four nonlinear control problems: velocity control under friction, one-link and two-link pendulum swing-up, and magnetic manipulation. The results show that the value functions yield well-performing policies and are compact, mathematically tractable, and easy to plug into other algorithms. This makes them potentially suitable for further analysis of the closed-loop system. A comparison with an alternative approach using neural networks shows that our method outperforms the neural network-based one.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑