Forward-PECVaR算法:CVaR SSP的精确计算
Forward-PECVaR Algorithm: Exact Evaluation for CVaR SSPs
摘要:
随机最短路径(SSP)问题模拟概率顺序决策问题,其中代理必须追求目标,同时最小化成本函数。由于概率动态,期望具有考虑风险的成本函数。条件风险价值(CVaR)是一种标准,它允许通过考虑更差轨迹的一部分$α$的期望来对任意风险水平进行建模。虽然最优策略是非马尔可夫的,但CVaR-SSP的解可以近似地用基于值迭代的算法找到,例如CVaR线性插值值迭代(CVaRVIQ)和CVaR分位数表示值迭代(CVaRVILI)。这些类型的解决方案取决于算法的参数,例如原子数和$α_0$(最小值$α$)。为了比较这些算法返回的策略,我们需要一种方法来准确评估CVaR-SSP的静态策略。虽然有一个算法来评估这些策略,但这只适用于具有统一成本的问题。在本文中,我们提出了一个新的算法,前向-PECVaR(ForPECVaR),准确地评估非均匀成本CVaR-SSP的平稳政策。我们评估经验CVaR值迭代算法,发现解决方案的质量与精确的解决方案相比,大约,和算法参数的质量和可扩展性的解决方案的影响。在两个域中的实验表明,重要的是使用小于$α$目标的$α_0$和足够数量的原子以获得良好的近似。
英文摘要:
The Stochastic Shortest Path (SSP) problem models probabilistic sequential-decision problems where an agent must pursue a goal while minimizing a cost function. Because of the probabilistic dynamics, it is desired to have a cost function that considers risk. Conditional Value at Risk (CVaR) is a criterion that allows modeling an arbitrary level of risk by considering the expectation of a fraction $α$ of worse trajectories. Although an optimal policy is non-Markovian, solutions of CVaR-SSP can be found approximately with Value Iteration based algorithms such as CVaR Value Iteration with Linear Interpolation (CVaRVIQ) and CVaR Value Iteration via Quantile Representation (CVaRVILI). These type of solutions depends on the algorithm's parameters such as the number of atoms and $α_0$ (the minimum $α$). To compare the policies returned by these algorithms, we need a way to exactly evaluate stationary policies of CVaR-SSPs. Although there is an algorithm that evaluates these policies, this only works on problems with uniform costs. In this paper, we propose a new algorithm, Forward-PECVaR (ForPECVaR), that evaluates exactly stationary policies of CVaR-SSPs with non-uniform costs. We evaluate empirically CVaR Value Iteration algorithms that found solutions approximately regarding their quality compared with the exact solution, and the influence of the algorithm parameters in the quality and scalability of the solutions. Experiments in two domains show that it is important to use an $α_0$ smaller than the $α$ target and an adequate number of atoms to obtain a good approximation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。