arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1405.3224math.STcs.LGstat.MLstat.TH
收藏

关于A/B测试的复杂性

On the Complexity of A/B Testing

LTCI, Télécom ParisTech & CNRS(LTCI,巴黎高等电信学院与法国国家科学研究中心);Institut de Mathématiques de Toulouse, Université Paul Sabatier(图卢兹数学研究所,保罗·萨巴捷大学)

Emilie Kaufmann, Olivier Cappé, Aurélien Garivier

摘要:

A/B测试是指在两个产生随机结果的选项中确定最佳选项的任务。我们提供了分布相关的下界的A/B测试的性能,提高了目前在固定置信度(或三角洲PAC)和固定预算设置的结果。当结果的分布是高斯分布时,我们证明了固定置信度和固定预算设置的复杂性是等价的,并且只有在方差相等的情况下,两种选择的均匀采样才是最优的。在共同方差的情况下,我们还提供了一个停止规则,终止速度比现有的固定置信度算法。在伯努利分布的情况下,我们表明,固定预算设置的复杂性是小于,固定的信心设置和均匀抽样的两个替代品-虽然不是最佳的-是可取的,在实践中结合适当的停止标准。

英文摘要:

A/B testing refers to the task of determining the best option among two alternatives that yield random outcomes. We provide distribution-dependent lower bounds for the performance of A/B testing that improve over the results currently available both in the fixed-confidence (or delta-PAC) and fixed-budget settings. When the distribution of the outcomes are Gaussian, we prove that the complexity of the fixed-confidence and fixed-budget settings are equivalent, and that uniform sampling of both alternatives is optimal only in the case of equal variances. In the common variance case, we also provide a stopping rule that terminates faster than existing fixed-confidence algorithms. In the case of Bernoulli distributions, we show that the complexity of fixed-budget setting is smaller than that of fixed-confidence setting and that uniform sampling of both alternatives -though not optimal- is advisable in practice when combined with an appropriate stopping criterion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑