Thompson sampling is a popular algorithm for solving multi-armed bandit problems, and has been applied in a wide range of applications, from website design to portfolio optimization. In such applications, however, the number of choices (or arms) $N$ can be large, and the data needed to make adaptive decisions require expensive experimentation. One is then faced with the constraint of experimenting on only a small subset of $K \ll N$ arms within each time period, which poses a problem for traditional Thompson sampling. We propose a new Thompson Sampling under Experimental Constraints (TSEC) method, which addresses this so-called "arm budget constraint". TSEC makes use of a Bayesian interaction model with effect hierarchy priors, to model correlations between rewards on different arms. This fitted model is then integrated within Thompson sampling, to jointly identify a good subset of arms for experimentation and to allocate resources over these arms. We demonstrate the effectiveness of TSEC in two problems with arm budget constraints. The first is a simulated website optimization study, where TSEC shows noticeable improvements over industry benchmarks. The second is a portfolio optimization application on industry-based exchange-traded funds, where TSEC provides more consistent and greater wealth accumulation over standard investment strategies.
Advertising Media and Target Audience Optimization via High-dimensional Bandits
Stanford University(斯坦福大学)
我们提出了一个数据驱动的算法,广告商可以使用它来自动化他们的数字广告活动在在线出版商。该算法使广告商能够搜索可用的目标受众和广告媒体,通过在线实验为其活动找到最佳组合。找到最佳受众广告组合的问题因许多独特的挑战而变得复杂,包括(a)需要积极探索以解决先前的不确定性并加速搜索有利可图的组合,(b)许多组合可供选择,从而产生高维搜索公式,以及(c)非常低的成功概率,通常只有百分之一的一小部分。我们的算法(命名为LRDL,是Logistic Regression with Debiased Lasso的首字母缩写)通过结合四个元素来解决这些挑战:用于主动探索的多臂强盗框架;用于处理高维的Lasso惩罚函数;处理Lasso引起的正则化偏差的内置去偏内核;以及用于促进跨臂交叉学习的结果的半参数回归模型。该算法被实现为汤普森采样器,据我们所知,它是第一个可以实际解决上述所有挑战的算法。真实数据和合成数据的仿真结果表明,该方法是有效的,并证明了其优越的性能对最近的高维bandit文献的几个基准。