arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1709.03465math.OC
收藏

弱耦合马尔可夫决策过程在线学习的收敛时间研究

Online Learning in Weakly Coupled Markov Decision Processes: A Convergence Time Study

Xiaohan Wei, Hao Yu, Michael J. Neely

摘要:

我们考虑多个由全局约束耦合的并行马尔可夫决策过程(MDP),其中时变目标和约束函数只有在做出决策后才能观察到。特别注意的是如何以及决策者可以执行$T$插槽,从任何状态开始,相比最好的可行的随机平稳策略事后。我们开发了一种新的分布式在线算法,每个MDP在观察从过去的信息计算出的乘数后,每个时隙做出自己的决定。虽然该方案比经典的在线学习环境更具挑战性,但该算法同时具有严格的$O(\sqrt{T})$遗憾和约束违反。为了获得这样的界,我们结合了几个新的成分,包括遍历性和混合时间界弱耦合MDP,一个新的遗憾分析在线约束优化,漂移分析排队过程,和扰动分析的基础上Farkas引理。

英文摘要:

We consider multiple parallel Markov decision processes (MDPs) coupled by global constraints, where the time varying objective and constraint functions can only be observed after the decision is made. Special attention is given to how well the decision maker can perform in $T$ slots, starting from any state, compared to the best feasible randomized stationary policy in hindsight. We develop a new distributed online algorithm where each MDP makes its own decision each slot after observing a multiplier computed from past information. While the scenario is significantly more challenging than the classical online learning context, the algorithm is shown to have a tight $O(\sqrt{T})$ regret and constraint violations simultaneously. To obtain such a bound, we combine several new ingredients including ergodicity and mixing time bound in weakly coupled MDPs, a new regret analysis for online constrained optimization, a drift analysis for queue processes, and a perturbation analysis based on Farkas' Lemma.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑