带切换成本的多臂老虎机的一种更快索引算法及计算研究
A faster index algorithm and a computational study for bandits with switching costs
AI总结:
针对带切换成本的多臂老虎机问题,提出解耦计算方法,将切换索引计算复杂度降至$n^2+O(n)$,实现四倍算术运算减少,实验证明策略近最优且显著优于基准。
摘要:
我们研究带有切换成本的难解多臂老虎机问题。Asawa和Teneketzis在[M. Asawa and D. Teneketzis. 1996. Multi-armed bandits with switching penalties. IEEE Trans. Automat. Control, 41 328-348]中引入了一个索引,该索引部分刻画了最优策略,为每个项目状态赋予一个“继续索引”(其Gittins index)和一个“切换索引”。他们提议将两者联合计算,视为具有$2 n$个状态的项目(当原项目有$n$个状态时)的Gittins index,导致$O(n^3)$算术运算量相比计算继续索引增加八倍。我们提出更快的解耦计算方法:第一阶段计算继续索引,第二阶段在最多$n^2 + O(n)$算术运算内计算切换索引,速度提升一个数量级,整体算术运算减少四倍,内存操作大幅降低。分析利用Asawa和Teneketzis索引是项目restless重构的边际生产力指数这一事实,采用作者引入的方法。大量计算实验表明,新算法显著加速运行,所得索引策略接近最优,且在随机生成的两项目和三项目实例中,相比基准Gittins index策略有显著增益。
英文摘要:
We address the intractable multi-armed bandit problem with switching costs, for which Asawa and Teneketzis introduced in [M. Asawa and D. Teneketzis. 1996. Multi-armed bandits with switching penalties. IEEE Trans. Automat. Control, 41 328-348] an index that partially characterizes optimal policies, attaching to each project state a ``continuation index'' (its Gittins index) and a ``switching index.'' They proposed to jointly compute both as the Gittins index of a project with $2 n$ states -- when the original project has $n$ states -- resulting in an eight-fold increase in $O(n^3)$ arithmetic operations relative to those to compute the continuation index. We present a faster decoupled computation method, which in a first stage computes the continuation index and then, in a second stage, computes the switching index an order of magnitude faster in at most $n^2 + O(n)$ arithmetic operations, achieving overall a four-fold reduction in arithmetic operations and substantially reduced memory operations. The analysis exploits the fact that the Asawa and Teneketzis index is the marginal productivity index of the project in its restless reformulation, using methods introduced by the author. Extensive computational experiments are reported, which demonstrate the dramatic runtime speedups achieved by the new algorithm, as well as the near-optimality of the resultant index policy and its substantial gains against the benchmark Gittins index policy across a wide range of randomly generated two- and three-project instances.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。