arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1702.07490cs.LG
收藏

基于并行算法竞赛的在线元学习

Online Meta-learning by Parallel Algorithm Competition

ATR Computational Neuroscience Laboratories(ATR计算神经科学实验室);Okinawa Institute of Science and Technology Graduate University(冲绳科学技术大学院大学)

Stefan Elfwing, Eiji Uchibe, Kenji Doya

摘要:

强化学习算法的效率主要取决于一些调节学习更新的元参数以及探索和利用之间的权衡。元参数的自适应是强化学习中的一个悬而未决的问题,最近随着深度强化学习在高维状态空间中的成功,这可以说已经成为一个问题。在诸如Atari 2600视频游戏的领域中的长学习时间使得执行适当的元参数值的全面搜索不可行。提出了并行算法竞赛在线元学习方法。在OMPAC方法中,并行运行强化学习算法的几个实例,其中元参数的初始值具有小的差异。在固定次数的事件之后,根据它们在手头任务中的表现来选择实例。在继续学习之前,以预定义的概率将高斯噪声添加到元参数。我们验证了OMPAC方法,通过改进随机SZ-俄罗斯方块和标准俄罗斯方块中较小的,10个$\times$ 10,板,分别为31%和84%,并通过改进三个Atari 2600游戏中的深度Sarsa($λ$)代理的结果62%或更多。实验还表明,OMPAC方法能够根据不同任务的学习进度来调整元参数。

英文摘要:

The efficiency of reinforcement learning algorithms depends critically on a few meta-parameters that modulates the learning updates and the trade-off between exploration and exploitation. The adaptation of the meta-parameters is an open question in reinforcement learning, which arguably has become more of an issue recently with the success of deep reinforcement learning in high-dimensional state spaces. The long learning times in domains such as Atari 2600 video games makes it not feasible to perform comprehensive searches of appropriate meta-parameter values. We propose the Online Meta-learning by Parallel Algorithm Competition (OMPAC) method. In the OMPAC method, several instances of a reinforcement learning algorithm are run in parallel with small differences in the initial values of the meta-parameters. After a fixed number of episodes, the instances are selected based on their performance in the task at hand. Before continuing the learning, Gaussian noise is added to the meta-parameters with a predefined probability. We validate the OMPAC method by improving the state-of-the-art results in stochastic SZ-Tetris and in standard Tetris with a smaller, 10$\times$10, board, by 31% and 84%, respectively, and by improving the results for deep Sarsa($λ$) agents in three Atari 2600 games by 62% or more. The experiments also show the ability of the OMPAC method to adapt the meta-parameters according to the learning progress in different tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑