arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.08233cs.LGcs.AIcs.NE
收藏

强化学习超参数优化的广义基于种群训练

Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning

Southern University of Science and Technology(南方科技大学)

Hui Bai, Ran Cheng

AI总结:

针对PBT偏爱高性能智能体而忽略探索潜力的问题,提出GPBT框架结合成对学习(PL)策略,通过识别性能差异全面指导表现不佳的智能体,在适应性和计算效率上显著提升,并在多个RL基准上超越传统PBT及其贝叶斯优化变体。

摘要:

超参数优化在机器学习领域起着关键作用。其重要性在强化学习(RL)中尤为突出,智能体不断与环境交互并适应环境,需要对其学习轨迹进行动态调整。为了适应这种动态性,引入了基于种群训练(PBT),利用同时学习的智能体种群的集体智慧。然而,PBT倾向于偏爱高性能智能体,可能忽略了处于重大进步边缘的智能体的探索潜力。为了减轻PBT的局限性,我们提出了广义基于种群训练(GPBT),这是一个旨在实现超参数适应增强粒度和灵活性的改进框架。作为GPBT的补充,我们进一步引入了成对学习(PL)。PL不仅关注精英智能体,还采用全面的成对策略来识别性能差异,并为表现不佳的智能体提供整体指导。通过整合GPBT和PL的能力,我们的方法在适应性和计算效率方面显著改进了传统PBT。在一系列RL基准上的严格实证评估证实,我们的方法不仅始终优于传统PBT,而且也优于其贝叶斯优化变体。

英文摘要:

Hyperparameter optimization plays a key role in the machine learning domain. Its significance is especially pronounced in reinforcement learning (RL), where agents continuously interact with and adapt to their environments, requiring dynamic adjustments in their learning trajectories. To cater to this dynamicity, the Population-Based Training (PBT) was introduced, leveraging the collective intelligence of a population of agents learning simultaneously. However, PBT tends to favor high-performing agents, potentially neglecting the explorative potential of agents on the brink of significant advancements. To mitigate the limitations of PBT, we present the Generalized Population-Based Training (GPBT), a refined framework designed for enhanced granularity and flexibility in hyperparameter adaptation. Complementing GPBT, we further introduce Pairwise Learning (PL). Instead of merely focusing on elite agents, PL employs a comprehensive pairwise strategy to identify performance differentials and provide holistic guidance to underperforming agents. By integrating the capabilities of GPBT and PL, our approach significantly improves upon traditional PBT in terms of adaptability and computational efficiency. Rigorous empirical evaluations across a range of RL benchmarks confirm that our approach consistently outperforms not only the conventional PBT but also its Bayesian-optimized variant.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑