优化马尔可夫决策过程中窗口平均收益的期望并提供保证
Optimising expectation with guarantees for window mean payoff in Markov decision processes
AI总结:
研究马尔可夫决策过程中带确定、几乎必然或概率阈值保证的窗口平均收益期望最大化策略综合,给出固定与有界窗口情形的复杂度及策略类型结论。
摘要:
窗口平均收益目标通过在沿无限路径滑动的有限窗口上计算平均收益,强化了经典平均收益目标。已有研究考虑了两种变体:在一种变体中,最大窗口长度是固定且给定的;而在另一种变体中,该长度并不固定,但要求有界。本文研究在马尔可夫决策过程中综合策略的问题,使窗口平均收益值在期望上最大化,同时保证该值高于某个阈值。我们针对三种不同类型的保证求解该综合问题:sure(确定保证,需要在最坏情况下满足,即面对对抗性环境时满足)、almost-sure(几乎必然保证,需要以概率1满足)和probabilistic(概率保证,需要至少以某个给定概率 p 满足)。我们证明,对于固定窗口平均收益目标,所有三个问题都属于 PTIME;而对于有界窗口平均收益目标,它们属于 NP ∩ coNP,因此与不带任何保证地最大化期望性能具有相同的复杂度。此外,我们证明,在带有确定保证和几乎必然保证的情况下,纯有限记忆策略足以最大化期望;而对于带概率保证的期望最大化,一般而言随机化策略是必要的。
英文摘要:
The window mean-payoff objective strengthens the classical mean-payoff objective by computing the mean-payoff over a finite window that slides along an infinite path. Two variants have been considered: in one variant, the maximum window length is fixed and given, while in the other, it is not fixed but is required to be bounded. In this paper, we look at the problem of synthesising strategies in Markov decision processes that maximise the window mean-payoff value in expectation, while also simultaneously guaranteeing that the value is above a certain threshold. We solve the synthesis problem for three different kinds of guarantees: sure (that needs to be satisfied in the worst-case, that is, for an adversarial environment), almost-sure (that needs to be satisfied with probability one), and probabilistic (that needs to be satisfied with at least some given probability $p$). We show that for fixed window mean-payoff objective, all the three problems are in $\mathsf{PTIME}$, while for bounded window mean-payoff objective, they are in $\mathsf{NP} \cap \mathsf{coNP}$, and thus have the same complexity as for maximising the expected performance without any guarantee. Moreover, we show that pure finite-memory strategies suffice for maximising the expectation with sure and almost-sure guarantees, whereas, for maximising expectation with a probabilistic guarantee, randomised strategies are necessary in general.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。