We consider Markov decision processes (MDPs) in which the transition probabilities and rewards belong to an uncertainty set parametrized by a collection of random variables. The probability distributions for these random parameters are unknown. The problem is to compute the probability to satisfy a temporal logic specification within any MDP that corresponds to a sample from these unknown distributions. In general, this problem is undecidable, and we resort to techniques from so-called scenario optimization. Based on a finite number of samples of the uncertain parameters, each of which induces an MDP, the proposed method estimates the probability of satisfying the specification by solving a finite-dimensional convex optimization problem. The number of samples required to obtain a high confidence on this estimate is independent from the number of states and the number of random parameters. Experiments on a large set of benchmarks show that a few thousand samples suffice to obtain high-quality confidence bounds with a high probability.
Certified Policy Verification and Synthesis for MDPs under Distributional Reach-avoidance Properties
Indian Institute of Technology Bombay(印度理工学院孟买分校);Institute of Science and Technology Austria (ISTA)(奥地利科学技术研究所);Lancaster University Leipzig(莱斯特兰卡斯特大学);Singapore Management University(新加坡管理大学)
1-2-3-Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization
Technical University of Munich(慕尼黑工业大学);Vali-e-Asr University of Rafsanjan(拉夫桑贾德瓦利阿斯尔大学);Masaryk University(马萨里克大学);Institute of Science and Technology(科学技术研究院)
Learning Algorithms for Verification of Markov Decision Processes
Masaryk University(马萨里克大学);IST Austria(奥地利科学技术研究所);Google LLC(谷歌有限责任公司);University of Oxford(牛津大学);Lancaster University Leipzig(莱比锡兰卡斯特大学);Rogers Communications(罗杰斯通信公司)