arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.07670cs.LG
收藏

通过深度网络的包围减少时间差值估计中的方差

Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks

University of California, Irvine(加州大学欧文分校);Carnegie Mellon University(卡内基梅隆大学);University of California, Berkeley(加州大学伯克利分校)

Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, Roy Fox

摘要:

在时间差强化学习算法中,值估计的方差会导致最大目标值的不稳定和高估。已经提出了许多算法来减少高估,包括最近的几种集成方法,但是没有一种算法通过解决估计方差作为高估的根本原因而在样本有效学习中取得成功。在本文中,我们提出了MeanQ,一个简单的集成方法,估计目标值作为集成手段。尽管它的简单性,MeanQ显示了显着的样本效率在Atari学习环境基准的实验。重要的是,我们发现大小为5的集合充分降低了估计方差以消除滞后目标网络,消除了它作为偏差的来源,并进一步提高了样本效率。我们直观和经验证明的设计选择均值Q,包括独立的经验抽样的必要性。在一组26个基准Atari环境中,MeanQ在16/26个环境中的100 K交互步骤上优于所有测试基线,包括最佳可用基线,日出,平均68%。MeanQ在21/26环境中的50万步的性能也优于Rainbow DQN,平均提高了49%,并在20万($\pm$ 100 K)交互步的情况下达到了人类水平的平均性能。我们的实现可以在https://github.com/indylab/MeanQ上获得。

英文摘要:

In temporal-difference reinforcement learning algorithms, variance in value estimation can cause instability and overestimation of the maximal target value. Many algorithms have been proposed to reduce overestimation, including several recent ensemble methods, however none have shown success in sample-efficient learning through addressing estimation variance as the root cause of overestimation. In this paper, we propose MeanQ, a simple ensemble method that estimates target values as ensemble means. Despite its simplicity, MeanQ shows remarkable sample efficiency in experiments on the Atari Learning Environment benchmark. Importantly, we find that an ensemble of size 5 sufficiently reduces estimation variance to obviate the lagging target network, eliminating it as a source of bias and further gaining sample efficiency. We justify intuitively and empirically the design choices in MeanQ, including the necessity of independent experience sampling. On a set of 26 benchmark Atari environments, MeanQ outperforms all tested baselines, including the best available baseline, SUNRISE, at 100K interaction steps in 16/26 environments, and by 68% on average. MeanQ also outperforms Rainbow DQN at 500K steps in 21/26 environments, and by 49% on average, and achieves average human-level performance using 200K ($\pm$100K) interaction steps. Our implementation is available at https://github.com/indylab/MeanQ.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑