arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.09999cs.LGcs.AI
收藏

Reward Centering(奖励中心化)

Reward Centering

University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究所);Meta AI

Abhishek Naik, Yi Wan, Manan Tomar, Richard S. Sutton

AI总结:

该研究提出奖励中心化方法,通过减去奖励经验均值优化持续性强化学习的折扣方法,在常用折扣因子下提升显著且不受奖励常数平移影响,还给出离策略场景的均值估计方案,可广泛适配各类强化学习算法。

摘要:

我们证明,用于求解持续性强化学习问题的折扣类方法,若通过减去奖励的经验平均值实现奖励中心化,性能可显著提升。在常用折扣因子下提升幅度明显,且折扣因子越接近1,提升效果越显著。此外我们发现,若问题的奖励被平移一个常数,标准方法性能会大幅下降,而采用奖励中心化的方法不受影响。在同策略(on-policy)场景下估计平均奖励十分简便;我们针对离策略(off-policy)场景提出了一种稍复杂的估计方法。奖励中心化是通用思路,我们认为几乎所有强化学习算法加入该机制后都能受益。

英文摘要:

We show that discounted methods for solving continuing reinforcement learning problems can perform significantly better if they center their rewards by subtracting out the rewards' empirical average. The improvement is substantial at commonly used discount factors and increases further as the discount factor approaches one. In addition, we show that if a problem's rewards are shifted by a constant, then standard methods perform much worse, whereas methods with reward centering are unaffected. Estimating the average reward is straightforward in the on-policy setting; we propose a slightly more sophisticated method for the off-policy setting. Reward centering is a general idea, so we expect almost every reinforcement-learning algorithm to benefit by the addition of reward centering.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑