多状态部分可观测不安宁Bands的索引性与转出策略
Indexability and Rollout Policy for Multi-State Partially Observable Restless Bandits
摘要：
具有部分可观测状态的不安分多臂强盗在通信系统、信息时代和推荐系统中有应用。本文研究了多状态部分可观测的不安宁强盗模型。我们考虑了三种不同的基于决策者可观察信息的模型-- 1）从强盗的行动中没有信息可观察2）来自强盗的完美信息仅对强盗的一个行动是可观察的，存在固定的重新启动状态，即，从所有其它状态到该状态的转换发生3）完美的状态信息可用于对Bandit的两个动作的决策者，并且对于两个动作存在两个重启状态。我们发展结构特性。我们还展示了模型2和模型3的阈值类型策略和可索引性。我们提出蒙特卡罗（MC）推出政策。在模型2的情况下，我们使用它来计算whittle指数。我们得到的集中界的价值函数的地平线长度和数量的轨迹MC推出政策。我们推导出模型3的显式指数公式。最后，我们描述了蒙特卡洛推出政策模型1时，很难显示可索引性。我们使用近视政策，蒙特卡罗推出政策和惠特尔指数政策的数值例子。我们观察到，蒙特卡洛推出政策是一个很好的竞争政策，近视。
英文摘要：
Restless multi-armed bandits with partially observable states has applications in communication systems, age of information and recommendation systems. In this paper, we study multi-state partially observable restless bandit models. We consider three different models based on information observable to decision maker -- 1) no information is observable from actions of a bandit 2) perfect information from bandit is observable only for one action on bandit, there is a fixed restart state, i.e., transition occurs from all other states to that state 3) perfect state information is available to decision maker for both actions on a bandit and there are two restart state for two actions. We develop the structural properties. We also show a threshold type policy and indexability for model 2 and 3. We present Monte Carlo (MC) rollout policy. We use it for whittle index computation in case of model 2. We obtain the concentration bound on value function in terms of horizon length and number of trajectories for MC rollout policy. We derive explicit index formula for model 3. We finally describe Monte Carlo rollout policy for model 1 when it is difficult to show indexability. We demonstrate the numerical examples using myopic policy, Monte Carlo rollout policy and Whittle index policy. We observe that Monte Carlo rollout policy is good competitive policy to myopic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
