arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.07525cs.LG
收藏

基于Polyak步长自适应的策略梯度增强方法

Enhancing Policy Gradient with the Polyak Step-Size Adaption

Stellantis(斯泰兰蒂斯集团);Télécom Paris(巴黎电信学院)

Yunxiang Li, Rui Yuan, Chen Fan, Mark Schmidt, Samuel Horváth, Robert M. Gower, Martin Takáč

AI总结:

针对强化学习中策略梯度算法对步长超参数敏感的问题,本文引入无需先验知识的Polyak自适应步长,解决了f*未知等适配问题,实验验证其能加快收敛、得到更稳定的策略。

摘要:

策略梯度是强化学习(RL)领域中应用广泛的基础算法。相较于其他RL算法,它以收敛性保证和稳定性著称,但实际应用常受限于对超参数(尤其是步长)的敏感性。本文提出将Polyak步长引入RL,无需先验知识即可自动调整步长。为使该方法适配RL场景,我们解决了若干问题,包括Polyak步长中f*未知的问题。此外,我们通过实验展示了Polyak步长在RL中的性能,证明其收敛速度更快,且能得到更稳定的策略。

英文摘要:

Policy gradient is a widely utilized and foundational algorithm in the field of reinforcement learning (RL). Renowned for its convergence guarantees and stability compared to other RL algorithms, its practical application is often hindered by sensitivity to hyper-parameters, particularly the step-size. In this paper, we introduce the integration of the Polyak step-size in RL, which automatically adjusts the step-size without prior knowledge. To adapt this method to RL settings, we address several issues, including unknown f* in the Polyak step-size. Additionally, we showcase the performance of the Polyak step-size in RL through experiments, demonstrating faster convergence and the attainment of more stable policies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑