arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1507.01585math.OCcs.SYeess.SY
收藏

有限时间 horizon 马尔可夫决策过程与状态约束

Finite-Horizon Markov Decision Processes with State Constraints

University of Texas at Austin(德克萨斯大学奥斯汀分校)

Mahmoud El Chamie, Behcet Acikmese

AI总结:

本文提出了一种高效算法,用于解决有限时间 horizon 约束马尔可夫决策过程中的非稳态随机策略问题,通过线性规划和对偶理论保证最优性。

摘要:

马尔可夫决策过程(MDPs)已被用于科学和工程中的许多决策问题。目标是合成最佳决策(动作选择)策略,以在给定的随机动态环境中最大化预期奖励(最小化成本)。在许多实际场景(多智能体系统、电信、排队等)中,决策问题可能有必须满足的状态约束,这导致了约束马尔可夫决策过程(CMDP)问题。在存在此类状态约束的情况下,最优策略可能非常难以描述。本文介绍了一种新的方法,用于寻找有限时间 horizon CMDP 的非稳态随机策略。提出了一种基于线性规划(LP)和对偶理论的高效算法,该算法给出了可行策略的凸集,并确保预期总奖励高于可计算的下界。所得到的决策策略是该无约束确定性 MDP 策略在该凸集上的投影。据我们所知,这是首个在状态约束 MDP 中提供生成有限时间 horizon 随机策略的高效算法并保证最优性的结果。还给出了一个自主代理群运行 MDP 的仿真示例,以展示所提出的 CMDP 解决方案算法。

英文摘要:

Markov Decision Processes (MDPs) have been used to formulate many decision-making problems in science and engineering. The objective is to synthesize the best decision (action selection) policies to maximize expected rewards (minimize costs) in a given stochastic dynamical environment. In many practical scenarios (multi-agent systems, telecommunication, queuing, etc.), the decision-making problem can have state constraints that must be satisfied, which leads to Constrained MDP (CMDP) problems. In the presence of such state constraints, the optimal policies can be very hard to characterize. This paper introduces a new approach for finding non-stationary randomized policies for finite-horizon CMDPs. An efficient algorithm based on Linear Programming (LP) and duality theory is proposed, which gives the convex set of feasible policies and ensures that the expected total reward is above a computable lower-bound. The resulting decision policy is a randomized policy, which is the projection of the unconstrained deterministic MDP policy on this convex set. To the best of our knowledge, this is the first result in state constrained MDPs to give an efficient algorithm for generating finite horizon randomized policies for CMDP with optimality guarantees. A simulation example of a swarm of autonomous agents running MDPs is also presented to demonstrate the proposed CMDP solution algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑