用Oracle引导的元强化学习求解真正大规模的预算约束单调POMDP
Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learning
AI总结：
针对大规模预算约束多组件单调POMDP求解难题，提出两步法：先用随机森林近似最优预算分配，再用Oracle引导的元训练PPO算法求解各单组件问题，实现可扩展性。
摘要：
单调部分可观察马尔可夫决策过程（POMDPs），其中系统状态持续下降直到执行恢复性操作，可以有效地用于建模序列修复问题。本文考虑求解预算约束的多组件单调POMDPs问题，其中有限的预算限制了恢复性操作的最大次数。对于大量组件，随着组件数量的增加，状态空间呈指数级增长，使用现有方法求解此类POMDP在计算上是不可行的。为了应对这一挑战，我们提出了一种两步方法。由于预算约束的多组件单调POMDP的各个组件仅通过共享预算相互关联，我们首先使用通过随机森林模型获得的每个组件POMDP最优值函数的近似值，来近似这些组件之间的最优预算分配。随后，我们引入一种由Oracle引导的元训练近端策略优化（PPO）算法，来求解每个独立的预算约束单组件单调POMDP。Oracle策略是通过对相应的单调马尔可夫决策过程（MDP）执行值迭代获得的。这种两步法在求解真正大规模的多组件单调POMDPs时提供了可扩展性。为了证明我们方法的有效性，我们考虑了一个真实的维护场景，该场景涉及一组智能体在维护预算内对行政大楼进行检查和修复。最后，我们对不同数量的组件进行了计算复杂度分析，以展示所提方法的可扩展性。
英文摘要：
Monotonic Partially Observable Markov Decision Processes (POMDPs), where the system state progressively decreases until a restorative action is performed, can be used to model sequential repair problems effectively. This paper considers the problem of solving budget-constrained multi-component monotonic POMDPs, where a finite budget limits the maximal number of restorative actions. For a large number of components, solving such a POMDP using current methods is computationally intractable due to the exponential growth in the state space with an increasing number of components. To address this challenge, we propose a two-step approach. Since the individual components of a budget-constrained multi-component monotonic POMDP are only connected via the shared budget, we first approximate the optimal budget allocation among these components using an approximation of each component POMDP's optimal value function which is obtained through a random forest model. Subsequently, we introduce an oracle-guided meta-trained Proximal Policy Optimization (PPO) algorithm to solve each of the independent budget-constrained single-component monotonic POMDPs. The oracle policy is obtained by performing value iteration on the corresponding monotonic Markov Decision Process (MDP). This two-step method provides scalability in solving truly massive multi-component monotonic POMDPs. To demonstrate the efficacy of our approach, we consider a real-world maintenance scenario that involves inspection and repair of an administrative building by a team of agents within a maintenance budget. Finally, we perform a computational complexity analysis for a varying number of components to show the scalability of the proposed approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
