RASR：结合EVaR与熵风险的风险厌恶型软鲁棒MDP
RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk
AI总结：
针对以往安全强化学习分别处理两类不确定性的局限，提出结合风险厌恶与软鲁棒的RASR框架，用时变风险水平的动态规划高效求解最优策略，实验验证其在多风险指标下有效缓解不确定性。
摘要：
以往安全强化学习（RL）研究分别针对动力学随机性（偶然不确定性）和模型不确定性（认知不确定性）的风险厌恶展开研究。我们提出并分析了一个新框架，用于在有限时域和折扣无限时域马尔可夫决策过程（MDP）中联合建模认知与偶然不确定性相关的风险，我们将这个结合风险厌恶与软鲁棒方法的框架称为RASR。我们证明，当使用EVaR（指数价值风险）或熵风险定义风险厌恶时，可通过一种具有时变风险水平的新型动态规划公式高效计算RASR中的最优策略。因此，即使在无限时域折扣设置下，最优风险厌恶策略也是确定性但随时间变化的。我们还表明，特定的RASR目标可简化为采用平均后验转移概率的风险厌恶强化学习。实验结果显示，我们的新算法在EVaR及其他标准风险指标衡量下，始终能有效缓解不确定性。
英文摘要：
Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to randomness in dynamics (aleatory) and to model uncertainty (epistemic) in isolation. We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and discounted infinite-horizon MDPs. We call this framework that combines Risk-Averse and Soft-Robust methods RASR. We show that when the risk-aversion is defined using either EVaR or the entropic risk, the optimal policy in RASR can be computed efficiently using a new dynamic program formulation with a time-dependent risk level. As a result, the optimal risk-averse policies are deterministic but time-dependent, even in the infinite-horizon discounted setting. We also show that particular RASR objectives reduce to risk-averse RL with mean posterior transition probabilities. Our empirical results show that our new algorithms consistently mitigate uncertainty as measured by EVaR and other standard risk measures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
