假设密度滤波Q学习
Assumed Density Filtering Q-learning
摘要：
虽然非策略时间差分（TD）方法由于其效率和简单的实现而被广泛用于强化学习，但它们的贝叶斯对应方法并没有被频繁地使用。一个原因是Bellman最优性方程中的非线性极大运算使得在价值函数上定义共轭分布变得困难。在本文中，我们介绍了一种新的贝叶斯方法，关闭政策TD方法，称为ADFQ，更新信念的状态动作值，Q，通过一个在线贝叶斯推理方法被称为假设密度过滤。我们制定了一个有效的封闭形式的解决方案，通过近似估计后验的Q信念的分析参数的值更新。信念中的不确定性度量不仅用于探索，而且还为考虑所有接下来可用动作的值更新提供了自然的正则化。随着Q信念的不确定性度量的降低，ADFQ收敛到Q学习，并改善了其他贝叶斯RL算法的常见缺点，例如计算复杂性。我们用一个神经网络扩展了ADFQ。我们的实证结果表明，ADFQ优于可比的算法在各种雅达利2600游戏，具有显着的改善，在高度随机域或域与一个大的动作空间。
英文摘要：
While off-policy temporal difference (TD) methods have widely been used in reinforcement learning due to their efficiency and simple implementation, their Bayesian counterparts have not been utilized as frequently. One reason is that the non-linear max operation in the Bellman optimality equation makes it difficult to define conjugate distributions over the value functions. In this paper, we introduce a novel Bayesian approach to off-policy TD methods, called as ADFQ, which updates beliefs on state-action values, Q, through an online Bayesian inference method known as Assumed Density Filtering. We formulate an efficient closed-form solution for the value update by approximately estimating analytic parameters of the posterior of the Q-beliefs. Uncertainty measures in the beliefs not only are used in exploration but also provide a natural regularization for the value update considering all next available actions. ADFQ converges to Q-learning as the uncertainty measures of the Q-beliefs decrease and improves common drawbacks of other Bayesian RL algorithms such as computational complexity. We extend ADFQ with a neural network. Our empirical results demonstrate that ADFQ outperforms comparable algorithms on various Atari 2600 games, with drastic improvements in highly stochastic domains or domains with a large action space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
