arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1408.2028cs.AI
收藏

树搜索的Bandit算法

Bandit Algorithms for Tree Search

Ecole Polytechnique(巴黎综合理工学院);INRIA Futurs Lille(法国国家信息与自动化研究所里尔未来分部)

Pierre-Arnuad Coquelin, Remi Munos

摘要:

基于Bandit的树搜索方法最近在应用于巨大的树时得到了普及,例如在围棋游戏中[6]。他们对树的有效探索使得能够快速地返回一个好的值,并且如果提供更多的时间,则可以提高精度。UCT算法[8]是一种基于上确界(UCB)[2]的树搜索方法,被认为局部适应树的有效光滑性。然而,我们表明,UCT是“过度乐观”在某种意义上,导致最坏情况下的遗憾,可能是非常差的。我们提出了替代强盗算法的树搜索。首先,修改UCT使用的置信度序列,在地平线深度指数缩放进行了分析。然后,我们考虑Flat-UCB执行的叶子,并提供了一个有限的遗憾界与高概率。然后,我们介绍并分析了一个光滑树的Bandit算法(BAST),该算法考虑了高置信度下执行次优分支的有效“切割”的回报的实际光滑性。最后,我们提出了一个增量树扩展,适用于当满树太大(可能是有限的),完全表示,并表明,具有很高的概率,只有最佳分支是无限发展。我们illustrate- trate这些方法上的一个连续函数的全局优化问题,噪声值。

英文摘要:

Bandit based methods for tree search have recently gained popularity when applied to huge trees, e.g. in the game of go [6]. Their efficient exploration of the tree enables to re- turn rapidly a good value, and improve preci- sion if more time is provided. The UCT algo- rithm [8], a tree search method based on Up- per Confidence Bounds (UCB) [2], is believed to adapt locally to the effective smoothness of the tree. However, we show that UCT is "over-optimistic" in some sense, leading to a worst-case regret that may be very poor. We propose alternative bandit algorithms for tree search. First, a modification of UCT us- ing a confidence sequence that scales expo- nentially in the horizon depth is analyzed. We then consider Flat-UCB performed on the leaves and provide a finite regret bound with high probability. Then, we introduce and analyze a Bandit Algorithm for Smooth Trees (BAST) which takes into account ac- tual smoothness of the rewards for perform- ing efficient "cuts" of sub-optimal branches with high confidence. Finally, we present an incremental tree expansion which applies when the full tree is too big (possibly in- finite) to be entirely represented and show that with high probability, only the optimal branches are indefinitely developed. We illus- trate these methods on a global optimization problem of a continuous function, given noisy values.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑