arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.05740cs.LGcs.AI
收藏

UGAE:一种新的非指数贴现方法

UGAE: A Novel Approach to Non-exponential Discounting

École Polytechnique(巴黎综合理工学院);CNRS(法国国家科学研究中心);Institut Polytechnique de Paris(巴黎理工学院);University of Rennes(雷恩大学);Inria(法国国家信息与自动化研究所);IRISA(法国雷恩信息系统与随机系统研究所)

Ariel Kwiatkowski, Vicky Kalogeiton, Julien Pettré, Marie-Paule Cani

摘要:

强化学习中的折扣机制决定了未来和现在奖励的相对重要性。虽然指数贴现在实践中被广泛使用,但与人类行为一致的非指数贴现方法通常用于创建类人代理。然而,非指数贴现方法不能直接应用于现代的政策行动者-批评者算法。为了解决这个问题,我们提出了通用广义优势估计(UGAE),它允许计算GAE优势值与任意折扣。此外,我们还引入了Beta加权贴现,这是指数和双曲线贴现之间的连续插值,以增加选择贴现方法的灵活性。为了展示UGAE的实用性,我们提供了各种贴现方法的属性分析。我们还通过实验表明,通过UGAE训练的具有非指数折扣的代理优于用Monte Carlo优势估计训练的变体。通过对各种折扣方法的分析和实验,我们证明了UGAE与Beta加权折扣在标准RL基准上的Monte Carlo基线上的优越性能。UGAE很简单,很容易集成到任何基于递归的算法中,作为标准递归GAE的替代。

英文摘要:

The discounting mechanism in Reinforcement Learning determines the relative importance of future and present rewards. While exponential discounting is widely used in practice, non-exponential discounting methods that align with human behavior are often desirable for creating human-like agents. However, non-exponential discounting methods cannot be directly applied in modern on-policy actor-critic algorithms. To address this issue, we propose Universal Generalized Advantage Estimation (UGAE), which allows for the computation of GAE advantage values with arbitrary discounting. Additionally, we introduce Beta-weighted discounting, a continuous interpolation between exponential and hyperbolic discounting, to increase flexibility in choosing a discounting method. To showcase the utility of UGAE, we provide an analysis of the properties of various discounting methods. We also show experimentally that agents with non-exponential discounting trained via UGAE outperform variants trained with Monte Carlo advantage estimation. Through analysis of various discounting methods and experiments, we demonstrate the superior performance of UGAE with Beta-weighted discounting over the Monte Carlo baseline on standard RL benchmarks. UGAE is simple and easily integrated into any advantage-based algorithm as a replacement for the standard recursive GAE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑