贝叶斯风险马尔可夫决策过程
Bayesian Risk Markov Decision Processes
摘要：
我们考虑有限时域马尔可夫决策过程的参数，如转移概率，是未知的，估计数据。流行的分布鲁棒方法来解决参数的不确定性，有时可能过于保守。在本文中，我们提出了一个新的配方，贝叶斯风险马尔可夫决策过程（BR-MDP），以解决参数的不确定性MDP，其中的风险功能适用于嵌套形式的预期总成本相对于贝叶斯后验分布的未知参数。拟议的公式提供了更灵活的风险态度对参数的不确定性，并考虑到在未来的时间阶段的数据的可用性。为了解决所提出的公式与条件风险价值（CVaR）的风险功能，我们提出了一个有效的近似算法，通过推导出一个解析近似的价值函数，并利用CVaR的凸性。我们证明了经验的BR-MDP制定和赌徒的投注问题和库存控制问题的算法的性能。
英文摘要：
We consider finite-horizon Markov Decision Processes where parameters, such as transition probabilities, are unknown and estimated from data. The popular distributionally robust approach to addressing the parameter uncertainty can sometimes be overly conservative. In this paper, we propose a new formulation, Bayesian risk Markov Decision Process (BR-MDP), to address parameter uncertainty in MDPs, where a risk functional is applied in nested form to the expected total cost with respect to the Bayesian posterior distribution of the unknown parameters. The proposed formulation provides more flexible risk attitutes towards parameter uncertainty and takes into account the availability of data in future times stages. To solve the proposed formulation with the conditional value-at-risk (CVaR) risk functional, we propose an efficient approximation algorithm by deriving an analytical approximation of the value function and utilizing the convexity of CVaR. We demonstrate the empirical performance of the BR-MDP formulation and proposed algorithms on a gambler's betting problem and an inventory control problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
