In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-value-at-risk (CVaR) objective, as opposed to a standard risk-neutral expectation. We refer to such problem as CVaR MDP. Our first contribution is to show that a CVaR objective, besides capturing risk sensitivity, has an alternative interpretation as expected cost under worst-case modeling errors, for a given error budget. This result, which is of independent interest, motivates CVaR MDPs as a unifying framework for risk-sensitive and robust decision making. Our second contribution is to present an approximate value-iteration algorithm for CVaR MDPs and analyze its convergence rate. To our knowledge, this is the first solution algorithm for CVaR MDPs that enjoys error guarantees. Finally, we present results from numerical experiments that corroborate our theoretical findings and show the practicality of our approach.
Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterion
Sun Yat-Sen University(中山大学);Stanford University(斯坦福大学)
CVaR(Conditional Value at Risk)是一种广泛应用于金融领域的风险度量方法。然而,动态优化CVaR是困难的,因为它不是一个标准的马尔可夫决策过程(MDP)和动态规划的原则失败。本文从灵敏度优化的角度研究了具有长期CVaR准则的无限时域离散时间MDP。通过引入一个伪CVaR度量,我们得到了一个CVaR差值公式,它量化了任意两种策略下长期CVaR的差值。推导了确定性策略的最优性。我们得到了CVaR的Bellman局部最优性方程,它是局部最优策略的充要条件,是全局最优策略的必要条件.为了提供更多的灵敏度信息,还推导了CVaR导数公式。在此基础上,我们提出了一种策略迭代型算法来有效地优化CVaR,并证明了该算法在混合策略空间中收敛于局部最优。我们进一步讨论了一些扩展,包括均值CVaR优化和最大化的CVaR。最后,我们进行了有关投资组合管理的数值实验,以证明主要结果。我们的工作可能揭示动态优化CVaR从灵敏度的角度来看。
ClauseLens: Clause-Grounded, CVaR-Constrained Reinforcement Learning for Trustworthy Reinsurance Pricing
Department of Applied Mathematics, University of California, Davis, CA, USA(加州大学戴维斯分校应用数学系);Wharton School of Business, University of Pennsylvania, Philadelphia, PA, USA(宾夕法尼亚大学沃顿商学院)
Measures of Variability for Risk-averse Policy Gradient
School of Computer Science, University of Waterloo(滑铁卢大学计算机科学学院);Vector Institute(矢量研究所);Department of Engineering Science, University of Oxford(牛津大学工程科学系);School of Computing Science, Simon Fraser University(西蒙菲莎大学计算科学学院)