arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1809.01220cs.AI
收藏

Vulcan:一种求解大机会约束MDP问题的Monte Carlo算法

Vulcan: A Monte Carlo Algorithm for Large Chance Constrained MDPs with Risk Bounding Functions

Massachusetts Institute of Technology(麻省理工学院)

Benjamin J Ayton, Brian C Williams

摘要:

机会约束马尔可夫决策过程在有限的失败概率下最大化回报,并且经常被应用于具有潜在危险结果或未知环境的规划。求解算法需要很强的逻辑性,或者仅限于具有数百万个状态的相对较小的问题,因为从给定状态采取的最佳行动取决于策略其余部分的失败概率,导致难以解决的耦合问题。在本文中,我们研究了CCMDP的一个推广,通过函数关系来权衡失败的概率对奖励。我们推导出一个约束,可以单独应用于每个国家的历史政策,并保证机会约束将得到满足。该方法对CCMDP中的状态进行了简化,从而可以有效地解决大型问题。然后,我们介绍Vulcan,它使用我们的约束,以适用于蒙特卡罗树搜索CCMDPs。Vulcan可以应用于无法生成整个状态空间的问题,并且必须以随时随地的方式返回策略。我们表明,火神和它的变种运行速度比线性规划方法快几十到几百倍,比基于启发式的方法快十倍以上,所有这些都不需要启发式,并返回平均次优的解决方案的顺序百分之几。最后,我们使用Vulcan来解决CCMDP中的机会约束策略,在3分钟内具有超过$10^{13}$的状态。

英文摘要:

Chance Constrained Markov Decision Processes maximize reward subject to a bounded probability of failure, and have been frequently applied for planning with potentially dangerous outcomes or unknown environments. Solution algorithms have required strong heuristics or have been limited to relatively small problems with up to millions of states, because the optimal action to take from a given state depends on the probability of failure in the rest of the policy, leading to a coupled problem that is difficult to solve. In this paper we examine a generalization of a CCMDP that trades off probability of failure against reward through a functional relationship. We derive a constraint that can be applied to each state history in a policy individually, and which guarantees that the chance constraint will be satisfied. The approach decouples states in the CCMDP, so that large problems can be solved efficiently. We then introduce Vulcan, which uses our constraint in order to apply Monte Carlo Tree Search to CCMDPs. Vulcan can be applied to problems where it is unfeasible to generate the entire state space, and policies must be returned in an anytime manner. We show that Vulcan and its variants run tens to hundreds of times faster than linear programming methods, and over ten times faster than heuristic based methods, all without the need for a heuristic, and returning solutions with a mean suboptimality on the order of a few percent. Finally, we use Vulcan to solve for a chance constrained policy in a CCMDP with over $10^{13}$ states in 3 minutes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑