We present a general framework for applying machine-learning algorithms to the verification of Markov decision processes (MDPs). The primary goal of these techniques is to improve performance by avoiding an exhaustive exploration of the state space. Our framework focuses on probabilistic reachability, which is a core property for verification, and is illustrated through two distinct instantiations. The first assumes that full knowledge of the MDP is available, and performs a heuristic-driven partial exploration of the model, yielding precise lower and upper bounds on the required probability. The second tackles the case where we may only sample the MDP, and yields probabilistic guarantees, again in terms of both the lower and upper bounds, which provides efficient stopping criteria for the approximation. The latter is the first extension of statistical model-checking for unbounded properties in MDPs. In contrast with other related approaches, we do not restrict our attention to time-bounded (finite-horizon) or discounted properties, nor assume any particular properties of the MDP. We also show how our techniques extend to LTL objectives. We present experimental results showing the performance of our framework on several examples.
Automated Experiment Design for Data-Efficient Verification of Parametric Markov Decision Processes
Department of Computer Science, University of Oxford(牛津大学计算机科学系);Department of Electrical Engineering, Eindhoven University of Technology(埃因霍温理工大学电气工程系)
Convex-Analytic View of Query Selection for Safety Verification of Markov Decision Processes
The Institute of Statistical Mathematics(统计数学研究所);Tohoku University(东北大学);RIKEN AIP(理化学研究所人工智能中心);CyberAgent;National Institute of Informatics(信息学国立研究所)
Marimba: A Tool for Verifying Properties of Hidden Markov Models
Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas(应用数学与系统研究所);University of Bristol(布里斯托尔大学);Bristol Robotics Laboratory(布里斯托尔机器人实验室);Universidad Nacional Autónoma de México(墨西哥国立自治大学)
Learning Algorithms for Verification of Markov Decision Processes
Masaryk University(马萨里克大学);IST Austria(奥地利科学技术研究所);Google LLC(谷歌有限责任公司);University of Oxford(牛津大学);Lancaster University Leipzig(莱比锡兰卡斯特大学);Rogers Communications(罗杰斯通信公司)