在平均奖励弱耦合MDP中实现$O(1/N)$最优性差距
Achieving an $O(1/N)$ Optimality Gap in Average-Reward Weakly-Coupled MDPs
浏览论文内容
中文总结 AI 辅助
本文针对平均奖励弱耦合MDP，提出一种非优先级策略，在特定条件下实现$O(1/N)$最优性差距，优于先前$1/\sqrt{N}$的结果。
中文摘要 AI 辅助
我们研究平均奖励弱耦合马尔可夫决策过程（WCMDPs），其中WCMDP由$N$个较小的MDP（称为臂）组成，这些臂共享多个每步预算约束。我们考虑臂具有相同模型参数、多个动作以及状态和动作相关成本的设置。对于restless bandits（RBs），这是WCMDPs的一个被充分研究的特例，先前的工作已经开发了在一般条件下实现$O(1/\sqrt{N})$最优性差距的策略，并进一步确定了策略可以实现优于$1/\sqrt{N}$最优性差距的条件。然而，对于一般的WCMDPs，没有先前的结果实现优于$1/\sqrt{N}$的最优性差距。在本文中，我们确定了与RBs类似的条件，在这些条件下可以实现优于$1/\sqrt{N}$的最优性差距，并设计了一种达到$O(1/N)$最优性差距的策略。值得注意的是，与先前基于推广优先级排序的方法不同，我们的策略不是基于优先级的，而是旨在诱导局部线性平均场动力学。
英文摘要
We study average-reward weakly-coupled Markov decision processes (WCMDPs), where a WCMDP consists of $N$ smaller MDPs, called arms, that share multiple per-step budget constraints. We consider the setting where the arms have identical model parameters, multiple actions, and state- and action-dependent costs. For restless bandits (RBs), a well-studied special case of WCMDPs, prior work has developed policies that achieve an $O(1/\sqrt{N})$ optimality gap under general conditions, and has further identified conditions under which policies can achieve a better-than-$1/\sqrt{N}$ optimality gap. However, for general WCMDPs, no prior result achieves an optimality gap better than $1/\sqrt{N}$. In this paper, we identify conditions analogous to those for RBs under which a better-than-$1/\sqrt{N}$ optimality gap is achievable, and design a policy that attains an $O(1/N)$ optimality gap. Notably, unlike prior approaches based on generalizing priority orderings, our policy is not priority-based but rather is designed to induce locally linear mean-field dynamics.
发表机构
- Georgia Institute of Technology（佐治亚理工学院）
- Harvard University（哈佛大学）
- University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
- Carnegie Mellon University（卡内基梅隆大学）
机构由 AI 辅助整理，请以论文原文为准。
