arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.09405cs.LGcs.AI
收藏

贝叶斯世代人口训练

Bayesian Generational Population-Based Training

University of Oxford(牛津大学);Amazon(亚马逊)

Xingchen Wan, Cong Lu, Jack Parker-Holder, Philip J. Ball, Vu Nguyen, Binxin Ru, Michael A. Osborne

摘要:

强化学习(RL)为训练能够在现实世界中自主交互的通用智能体提供了潜力。然而,一个关键的限制是RL算法对核心超参数和网络架构选择的脆弱性。此外,非平稳性,如不断变化的训练数据和增加的代理复杂性意味着不同的超参数和架构可能在不同的训练点是最佳的。这激发了AutoRL,这是一类寻求自动化这些设计选择的方法。AutoRL方法的一个突出类别是基于人口的训练(PBT),它在几个大规模设置中具有令人印象深刻的性能。在本文中,我们介绍了两个新的创新的PBT式的方法。首先,我们采用基于信赖域的贝叶斯优化,使高维混合超参数搜索空间的全覆盖。其次,我们证明了使用分代方法,我们也可以在一次训练中同时学习架构和超参数。利用新的高度并行化的Brax物理引擎,我们证明了这些创新带来了巨大的性能提升,在动态学习整个配置的同时显著优于调整后的基线。代码可在https://github.com/xingchenwan/bgpbt上获得。

英文摘要:

Reinforcement learning (RL) offers the potential for training generally capable agents that can interact autonomously in the real world. However, one key limitation is the brittleness of RL algorithms to core hyperparameters and network architecture choice. Furthermore, non-stationarities such as evolving training data and increased agent complexity mean that different hyperparameters and architectures may be optimal at different points of training. This motivates AutoRL, a class of methods seeking to automate these design choices. One prominent class of AutoRL methods is Population-Based Training (PBT), which have led to impressive performance in several large scale settings. In this paper, we introduce two new innovations in PBT-style methods. First, we employ trust-region based Bayesian Optimization, enabling full coverage of the high-dimensional mixed hyperparameter search space. Second, we show that using a generational approach, we can also learn both architectures and hyperparameters jointly on-the-fly in a single training run. Leveraging the new highly parallelizable Brax physics engine, we show that these innovations lead to large performance gains, significantly outperforming the tuned baseline while learning entire configurations on the fly. Code is available at https://github.com/xingchenwan/bgpbt.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑