arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.04157eess.SYcs.SY
收藏

风险敏感指数代价MDP的策略梯度算法

A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP

Mehrdad Moharrami, Yashaswini Murthy, Arghyadip Roy, R. Srikant

摘要:

我们研究了风险敏感的指数成本MDP公式,并开发了一个基于概率的梯度算法,以找到与一组参数化的政策相关的成本的平稳点。我们推导出一个公式,可以用来计算从(状态,动作,成本)信息收集的样本路径的MDP为每个固定的参数化政策的政策梯度。与传统的平均成本问题不同,标准的随机近似理论不能用于利用这个公式。为了解决这个问题,我们引入了一个截断和光滑版本的风险敏感成本,并表明,这个新的成本标准可以用来近似的风险敏感成本和梯度一致的一些温和的假设。然后,我们开发了一个基于概率的梯度算法,以尽量减少平滑截断估计的风险敏感成本和条件下,一系列的截断可以用来解决原来的,未截断的成本问题。

英文摘要:

We study the risk-sensitive exponential cost MDP formulation and develop a trajectory-based gradient algorithm to find the stationary point of the cost associated with a set of parameterized policies. We derive a formula that can be used to compute the policy gradient from (state, action, cost) information collected from sample paths of the MDP for each fixed parameterized policy. Unlike the traditional average-cost problem, standard stochastic approximation theory cannot be used to exploit this formula. To address the issue, we introduce a truncated and smooth version of the risk-sensitive cost and show that this new cost criterion can be used to approximate the risk-sensitive cost and its gradient uniformly under some mild assumptions. We then develop a trajectory-based gradient algorithm to minimize the smooth truncated estimation of the risk-sensitive cost and derive conditions under which a sequence of truncations can be used to solve the original, untruncated cost problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑