arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.17833cs.LGcs.AI
收藏

深度强化学习中的优化器重置:一项实证研究

Resetting the Optimizer in Deep RL: An Empirical Study

Amazon(亚马逊);Technion(以色列理工学院)

Kavosh Asadi, Rasool Fakoor, Shoham Sabach

AI总结:

本研究针对深度强化学习中优化器内部参数因损失函数变化而受污染的问题,提出重置优化器内部参数的简单方法,并与Rainbow算法结合,在Atari基准上显著提升了性能。

摘要:

我们聚焦于深度强化学习中近似最优价值函数的任务。这一迭代过程由求解一系列优化问题组成,其中损失函数在每次迭代中都会发生变化。解决这一系列问题的常见方法是采用随机梯度下降算法的现代变体,如Adam。这些优化器维护自身的内部参数,例如梯度的一阶矩和二阶矩的估计,并随时间更新它们。因此,先前迭代中获得的信息被用于解决当前迭代的优化问题。我们证明,这可能会污染矩估计,因为优化景观在每次迭代之间可能任意变化。为了对冲这种负面影响,一个简单的想法是在开始新迭代时重置优化器的内部参数。我们通过将各种优化器与Rainbow算法结合使用,对重置这一想法进行了实证研究。我们证明,这一简单修改显著提升了深度强化学习在Atari基准上的性能。

英文摘要:

We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second-order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑