arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.06276cs.LGcs.AI
收藏

基线在策略梯度优化中的作用

The Role of Baselines in Policy Gradient Optimization

Google Research(谷歌研究院);Mila(米拉研究所);McGill University(麦吉尔大学);University of Montreal(蒙特利尔大学);DeepMind(深度思维);Amii(阿尔伯塔机器智能研究所);University of Alberta(阿尔伯塔大学)

Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, Dale Schuurmans

摘要:

研究了基线对策略内随机策略梯度优化的影响,弥补了策略优化方法理论与实践之间的差距。我们的第一个贡献是表明,\emph{state value}基线允许对政策的随机\emph{natural}策略梯度(NPG)收敛到一个全局最优的政策在$O(1/t)$率,这是以前不知道的。该分析依赖于两个新的发现:NPG更新的预期进度满足非均匀的NPG不等式的随机版本,并且在概率为1的情况下,状态值基线防止最佳动作的概率消失,从而确保充分的探索。重要的是,这些结果提供了一个新的理解随机策略梯度中的基线的作用:通过显示,自然策略梯度估计的方差仍然无界或没有基线,我们发现,方差减少\emph{cannot}解释他们在这种情况下的效用。相反,分析显示,价值基线的主要影响是对\textbf{reduce the aggressiveness of the updates}的影响,而不是它们的方差。也就是说,我们证明了一个有限的方差是\emph{not necessary}几乎肯定收敛的随机NPG,而控制更新的积极性是必要的和充分的。进一步的实验结果验证了这些理论发现。

英文摘要:

We study the effect of baselines in on-policy stochastic policy gradient optimization, and close the gap between the theory and practice of policy optimization methods. Our first contribution is to show that the \emph{state value} baseline allows on-policy stochastic \emph{natural} policy gradient (NPG) to converge to a globally optimal policy at an $O(1/t)$ rate, which was not previously known. The analysis relies on two novel findings: the expected progress of the NPG update satisfies a stochastic version of the non-uniform Łojasiewicz (NŁ) inequality, and with probability 1 the state value baseline prevents the optimal action's probability from vanishing, thus ensuring sufficient exploration. Importantly, these results provide a new understanding of the role of baselines in stochastic policy gradient: by showing that the variance of natural policy gradient estimates remains unbounded with or without a baseline, we find that variance reduction \emph{cannot} explain their utility in this setting. Instead, the analysis reveals that the primary effect of the value baseline is to \textbf{reduce the aggressiveness of the updates} rather than their variance. That is, we demonstrate that a finite variance is \emph{not necessary} for almost sure convergence of stochastic NPG, while controlling update aggressiveness is both necessary and sufficient. Additional experimental results verify these theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑