arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv cs/0703062cs.LG
收藏

树搜索的Bandit算法

Bandit Algorithms for Tree Search

Pierre-Arnaud Coquelin, Rémi Munos

摘要:

用于树搜索的基于Bandit的方法最近在应用于巨大的树时(例如在围棋游戏中)已经流行起来(Gelly等人,2006年)。UCT算法(Kocsis和Szepesvari,2006)是一种基于上置信界限(UCB)的树搜索方法(Auer等人,2002年),被认为是适应当地的有效平滑的树。然而,我们发现UCT在某些情况下过于“乐观”,导致遗憾O(exp(exp(D),其中D是树的深度。我们提出了替代的强盗算法树搜索。首先,使用置信度序列的UCT的修改,该置信度序列与水平深度呈指数级缩放被证明具有遗憾O(2^D \sqrt{n}),但不适应树中可能的平滑度。然后,我们分析了Flat-UCB执行的叶子,并提供了一个有限的遗憾界的概率很高。然后,我们引入了一个基于UCB的Bandit算法的光滑树,它考虑到实际的光滑度的奖励执行有效的“切割”的次优分支具有高的信心。最后,我们提出了一个增量树搜索版本,适用于当满树太大(可能是无限的),完全表示,并显示,具有很高的概率,基本上只有最佳分支是无限发展。我们说明了这些方法的Lipschitz函数的全局优化问题,噪声数据。

英文摘要:

Bandit based methods for tree search have recently gained popularity when applied to huge trees, e.g. in the game of go (Gelly et al., 2006). The UCT algorithm (Kocsis and Szepesvari, 2006), a tree search method based on Upper Confidence Bounds (UCB) (Auer et al., 2002), is believed to adapt locally to the effective smoothness of the tree. However, we show that UCT is too ``optimistic'' in some cases, leading to a regret O(exp(exp(D))) where D is the depth of the tree. We propose alternative bandit algorithms for tree search. First, a modification of UCT using a confidence sequence that scales exponentially with the horizon depth is proven to have a regret O(2^D \sqrt{n}), but does not adapt to possible smoothness in the tree. We then analyze Flat-UCB performed on the leaves and provide a finite regret bound with high probability. Then, we introduce a UCB-based Bandit Algorithm for Smooth Trees which takes into account actual smoothness of the rewards for performing efficient ``cuts'' of sub-optimal branches with high confidence. Finally, we present an incremental tree search version which applies when the full tree is too big (possibly infinite) to be entirely represented and show that with high probability, essentially only the optimal branches is indefinitely developed. We illustrate these methods on a global optimization problem of a Lipschitz function, given noisy data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑