arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1806.01825cs.AI
收藏

高维状态空间中规划形状对Dyna型规划的影响

The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces

University of Alberta(阿尔伯塔大学);Franklin & Marshall College(富兰克林与马歇尔学院)

G. Zacharias Holland, Erin J. Talvitie, Michael Bowling

摘要:

Dyna是基于模型的强化学习(MBRL)的基本方法,它在在线环境中交织规划、行动和学习。在Dyna最典型的应用中,动态模型用于从代理的历史中选择的起始状态生成单步转换,这些转换用于更新代理的价值函数或策略,就像它们是真实的经验一样。在这项工作中,一步Dyna应用于街机学习环境(ALE)的几个游戏。我们发现,基于模型的更新提供了令人惊讶的好处比简单地执行更多的更新与代理的现有经验,即使在使用一个完美的模型。我们假设,为了从规划中获得最大的收益,必须使用模型来产生不熟悉的经验。为了测试这一点,我们在Dyna的多个不同的具体实例中试验了规划的“形状”,执行更少、更长的推出,而不是许多短的推出。我们发现,规划形状对Dyna的功效有着深远的影响,无论是完美的还是学习的模型。除了这些研究结果,关于Dyna一般,我们的研究结果代表,据我们所知,第一次学习的动力学模型已成功地用于规划在ALE,这表明Dyna可能是一个可行的方法MBRL在ALE和其他高维问题。

英文摘要:

Dyna is a fundamental approach to model-based reinforcement learning (MBRL) that interleaves planning, acting, and learning in an online setting. In the most typical application of Dyna, the dynamics model is used to generate one-step transitions from selected start states from the agent's history, which are used to update the agent's value function or policy as if they were real experiences. In this work, one-step Dyna was applied to several games from the Arcade Learning Environment (ALE). We found that the model-based updates offered surprisingly little benefit over simply performing more updates with the agent's existing experience, even when using a perfect model. We hypothesize that to get the most from planning, the model must be used to generate unfamiliar experience. To test this, we experimented with the "shape" of planning in multiple different concrete instantiations of Dyna, performing fewer, longer rollouts, rather than many short rollouts. We found that planning shape has a profound impact on the efficacy of Dyna for both perfect and learned models. In addition to these findings regarding Dyna in general, our results represent, to our knowledge, the first time that a learned dynamics model has been successfully used for planning in the ALE, suggesting that Dyna may be a viable approach to MBRL in the ALE and other high-dimensional problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑