arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.09930cs.AIcs.SYeess.SY
收藏

具有定量目标的随机博弈上值迭代的停止准则

Stopping Criteria for Value Iteration on Stochastic Games with Quantitative Objectives

Institute of Science and Technology Austria(奥地利科学技术研究所);Masaryk University(马萨里克大学);Technical University of Munich(慕尼黑工业大学)

Jan Křetínský, Tobias Meggendorfer, Maximilian Weininger

AI总结:

本文针对总奖励与平均收益随机博弈上的值迭代,提出首批停止准则及任意时间算法,并通过归约MDP与直接SG两种方案统一现有方法。

摘要:

Markov决策过程(MDP)和随机博弈(SG)的一种经典求解技术是值迭代(VI)。由于其良好的实际性能,这种近似方法通常比精确技术更受青睐,尽管直到最近仍无法给出关于结果不精确性的实用界。因此,即使是最常用的模型检测器也可能返回任意错误的结果。在过去十年中,不同工作针对各种情形推导了停止准则,用于指示精度何时达到期望水平,特别是针对具有可达性、总奖励和平均收益的MDP,以及具有可达性的SG。在本文中,我们首次为具有总奖励和平均收益的SG上的VI提供停止准则,从而得到这些情形下的首批任意时间算法。为此,我们以两种形式给出解决方案:第一种通过归约到MDP情形,第二种直接在SG上进行。前者更简单,并能自动利用MDP方面的任何进展。后者允许更多局部计算,旨在获得更好的实际效率。我们的解决方案统一了前面提到的MDP和SG方法及其底层思想。为实现这一点,我们分离了特定于目标的子程序,并识别了独立于目标的概念。这些结构概念虽然出奇地简单,却构成了统一解决方案的核心本质。

英文摘要:

A classic solution technique for Markov decision processes (MDP) and stochastic games (SG) is value iteration (VI). Due to its good practical performance, this approximative approach is typically preferred over exact techniques, even though no practical bounds on the imprecision of the result could be given until recently. As a consequence, even the most used model checkers could return arbitrarily wrong results. Over the past decade, different works derived stopping criteria, indicating when the precision reaches the desired level, for various settings, in particular MDP with reachability, total reward, and mean payoff, and SG with reachability. In this paper, we provide the first stopping criteria for VI on SG with total reward and mean payoff, yielding the first anytime algorithms in these settings. To this end, we provide the solution in two flavours: First through a reduction to the MDP case and second directly on SG. The former is simpler and automatically utilizes any advances on MDP. The latter allows for more local computations, heading towards better practical efficiency. Our solution unifies the previously mentioned approaches for MDP and SG and their underlying ideas. To achieve this, we isolate objective-specific subroutines as well as identify objective-independent concepts. These structural concepts, while surprisingly simple, form the very essence of the unified solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑