arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.11151cs.LGcs.AIstat.ML
收藏

逻辑Q学习

Logistic Q-Learning

Universitat Pompeu Fabra(庞培法布拉大学);ETH Zürich(苏黎世联邦理工学院)

Joan Bas-Serrano, Sebastian Curi, Andreas Krause, Gergely Neu

摘要:

我们提出了一种新的强化学习算法,该算法源自MDP中最优控制的正则化线性规划公式。该方法与Peters等人(2010)的经典相对熵策略搜索(REPS)算法密切相关,关键区别在于我们的方法引入了一个Q函数,可以实现高效精确的无模型实现。我们的算法(称为QREPS)的主要特点是一个凸的损失函数的政策评估,作为一个理论上健全的替代广泛使用的平方贝尔曼误差。我们提供了一个实用的鞍点优化方法,最大限度地减少这个损失函数,并提供了一个错误传播分析,涉及到个人更新的输出策略的性能的质量。最后,我们证明了我们的方法在一系列基准问题的有效性。

英文摘要:

We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The method is closely related to the classic Relative Entropy Policy Search (REPS) algorithm of Peters et al. (2010), with the key difference that our method introduces a Q-function that enables efficient exact model-free implementation. The main feature of our algorithm (called QREPS) is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error. We provide a practical saddle-point optimization method for minimizing this loss function and provide an error-propagation analysis that relates the quality of the individual updates to the performance of the output policy. Finally, we demonstrate the effectiveness of our method on a range of benchmark problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑