Thompson抽样引导的欺骗环境线上随机搜索及其在寻根问题中的应用
Thompson Sampling Guided Stochastic Searching on the Line for Deceptive Environments with Applications to Root-Finding Problems
摘要:
多臂强盗问题通过一个简单而有效的机制,形成了解决广泛的在线随机优化问题的基础。一个简单的问题是一个赌徒反复从N个老虎机臂中抽出一个,随机获得奖励。然后,通过仔细平衡奖励探索和奖励利用,将奖励概率的学习与奖励最大化相结合。在本文中,我们解决了一个特别有趣的变种的多臂土匪问题,称为{ \it随机点定位(SPL)问题}。在这里,赌徒仅被告知最佳手臂(点)是位于所拉手臂的“左边”还是“右边”,反馈是错误的概率为$1-π$。因此,该公式在具有{ \it信息}和{ \it欺骗}反馈的连续动作空间中捕获优化。为了解决这类问题,我们制定了一个紧凑的和可扩展的贝叶斯表示的解决方案空间,同时捕获的最佳手臂的位置以及接收正确的反馈的概率。我们还介绍了汤普森抽样指导随机点定位(TS-SPL)计划平衡勘探开发。通过学习$π$,TS-SPL还支持{ \it欺骗}环境,这些环境在最优臂的方向上撒谎。这反过来又允许我们解决基本的随机根查找(SRF)问题。实证结果表明,我们的计划处理欺骗性和信息的环境,显着优于竞争的算法都SRF和SPL。
英文摘要:
The multi-armed bandit problem forms the foundation for solving a wide range of on-line stochastic optimization problems through a simple, yet effective mechanism. One simply casts the problem as a gambler that repeatedly pulls one out of N slot machine arms, eliciting random rewards. Learning of reward probabilities is then combined with reward maximization, by carefully balancing reward exploration against reward exploitation. In this paper, we address a particularly intriguing variant of the multi-armed bandit problem, referred to as the {\it Stochastic Point Location (SPL) Problem}. The gambler is here only told whether the optimal arm (point) lies to the "left" or to the "right" of the arm pulled, with the feedback being erroneous with probability $1-π$. This formulation thus captures optimization in continuous action spaces with both {\it informative} and {\it deceptive} feedback. To tackle this class of problems, we formulate a compact and scalable Bayesian representation of the solution space that simultaneously captures both the location of the optimal arm as well as the probability of receiving correct feedback. We further introduce the accompanying Thompson Sampling guided Stochastic Point Location (TS-SPL) scheme for balancing exploration against exploitation. By learning $π$, TS-SPL also supports {\it deceptive} environments that are lying about the direction of the optimal arm. This, in turn, allows us to solve the fundamental Stochastic Root Finding (SRF) Problem. Empirical results demonstrate that our scheme deals with both deceptive and informative environments, significantly outperforming competing algorithms both for SRF and SPL.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。