arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.02827cs.LGcs.AIstat.ML
收藏

马氏一致风险下策略梯度的收敛性与最优性

On the Convergence and Optimality of Policy Gradient for Markov Coherent Risk

Carnegie Mellon University(卡内基梅隆大学);Purdue University(普渡大学)

Audrey Huang, Liu Leqi, Zachary C. Lipton, Kamyar Azizzadenesheli

摘要:

为了在强化学习中对风险厌恶进行建模,一种新兴的研究方法采用了熟悉的算法来优化相干风险泛函,这是一个包括条件风险价值(CVaR)的类。由于在马尔可夫决策过程中优化相关风险是困难的,最近的工作往往集中在马尔可夫相关风险(MCR),一个时间一致的代理。虽然,策略梯度(PG)的更新已得出这一目标,它仍然不清楚(i)PG是否找到一个全球最佳的MCR;(ii)如何估计梯度在一个易于处理的方式。在本文中,我们证明,在一般情况下,MCR目标(不同于预期收益)不是梯度主导的,驻点一般不能保证是全局最优的。此外,我们提出了一个严格的上界的次优学习政策,其依赖于目标的非线性和风险规避程度的特点。解决(ii),我们提出了一个实际的实现PG,使用状态分布重新加权,以克服以前的限制。通过实验,我们证明了当最优性差距很小时,PG可以学习风险敏感的政策。然而,我们发现,具有较大次优差距的实例是丰富的,易于构建,概述了未来研究的一个重要挑战。

英文摘要:

In order to model risk aversion in reinforcement learning, an emerging line of research adapts familiar algorithms to optimize coherent risk functionals, a class that includes conditional value-at-risk (CVaR). Because optimizing the coherent risk is difficult in Markov decision processes, recent work tends to focus on the Markov coherent risk (MCR), a time-consistent surrogate. While, policy gradient (PG) updates have been derived for this objective, it remains unclear (i) whether PG finds a global optimum for MCR; (ii) how to estimate the gradient in a tractable manner. In this paper, we demonstrate that, in general, MCR objectives (unlike the expected return) are not gradient dominated and that stationary points are not, in general, guaranteed to be globally optimal. Moreover, we present a tight upper bound on the suboptimality of the learned policy, characterizing its dependence on the nonlinearity of the objective and the degree of risk aversion. Addressing (ii), we propose a practical implementation of PG that uses state distribution reweighting to overcome previous limitations. Through experiments, we demonstrate that when the optimality gap is small, PG can learn risk-sensitive policies. However, we find that instances with large suboptimality gaps are abundant and easy to construct, outlining an important challenge for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑