arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.10311cs.LGcs.AI
收藏

了解每个步骤不同重放次数的影响

Understanding the effect of varying amounts of replay per step

University of Alberta(阿尔伯塔大学)

Animesh Kumar Paul, Videh Raj Nema

摘要:

基于模型的强化学习使用模型进行规划,其中可以通过使用更多的计算来改进代理的预测和策略,而无需来自环境的额外数据,从而提高样本效率。然而,学习模型的准确估计是困难的。随后,自然的问题是,我们是否可以获得与无模型方法规划类似的好处。经验重放是许多无模型算法的重要组成部分,通过提供一种机制来存储过去的经验,以便在梯度计算过程中进一步重用,从而实现样本高效学习和稳定性。先前的工作已经建立了模型之间的联系,并通过与后者的规划经验重放。这涉及到在每个步骤中增加对小批量进行采样并用于更新的次数(每步的重放量)。我们试图利用这种联系,在一个著名的无模型算法:深度Q网络(DQN)在山地车环境中,对每一步不同的重放量的影响进行系统的研究。我们的经验表明,增加重放提高DQN的样本效率,减少其性能的变化,并使其更强大的超参数的变化。总而言之,这朝着更好的部署算法迈出了一步。

英文摘要:

Model-based reinforcement learning uses models to plan, where the predictions and policies of an agent can be improved by using more computation without additional data from the environment, thereby improving sample efficiency. However, learning accurate estimates of the model is hard. Subsequently, the natural question is whether we can get similar benefits as planning with model-free methods. Experience replay is an essential component of many model-free algorithms enabling sample-efficient learning and stability by providing a mechanism to store past experiences for further reuse in the gradient computational process. Prior works have established connections between models and experience replay by planning with the latter. This involves increasing the number of times a mini-batch is sampled and used for updates at each step (amount of replay per step). We attempt to exploit this connection by doing a systematic study on the effect of varying amounts of replay per step in a well-known model-free algorithm: Deep Q-Network (DQN) in the Mountain Car environment. We empirically show that increasing replay improves DQN's sample efficiency, reduces the variation in its performance, and makes it more robust to change in hyperparameters. Altogether, this takes a step toward a better algorithm for deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑