arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.14628cs.LGstat.ME
收藏

快速且可扩展的贝叶斯 AB 测试

Rapid and Scalable Bayesian AB Testing

Apple(苹果公司)

Srivas Chennu, Andrew Maher, Christian Pangerl, Subash Prabanantham, Jae Hyeon Bae, Jamie Martin, Bud Goswami

AI总结:

本文提出一种基于分层贝叶斯估计的 AB 测试方法,利用因素间相关性提高统计功效,支持序贯检验与渐进式提前停止,并能从过往测试中提取复合全局经验以加速未来测试。

摘要:

AB 测试帮助业务运营者进行决策,被视为从数据中学习以改善数字用户体验的黄金标准方法。然而,从业者需求与常用于 AB 测试分析的统计假设检验方法所施加的约束之间通常存在差距。这些差距包括在包含多因素的多变量设计中缺乏统计功效、因素间的相关性、提前停止所需的序贯检验需求,以及无法汇集过往测试的知识。在此,我们提出一种应用分层贝叶斯估计的解决方案以解决上述局限性。与当前的序贯 AB 测试方法相比,我们通过利用因素间的相关性提高了统计功效,实现了序贯检验和渐进式提前停止,且不会带来过高的假阳性风险。我们还展示了如何扩展该方法以从过往 AB 测试中提取复合全局经验,从而加速未来测试。我们以坚实的理论框架为基础,阐明了分层估计的价值。我们通过数值模拟和大量真实世界的 AB 测试展示了其效用。这些结果共同突显了我们的方法在科技行业统计推断中的实用价值。

英文摘要:

AB testing aids business operators with their decision making, and is considered the gold standard method for learning from data to improve digital user experiences. However, there is usually a gap between the requirements of practitioners, and the constraints imposed by the statistical hypothesis testing methodologies commonly used for analysis of AB tests. These include the lack of statistical power in multivariate designs with many factors, correlations between these factors, the need of sequential testing for early stopping, and the inability to pool knowledge from past tests. Here, we propose a solution that applies hierarchical Bayesian estimation to address the above limitations. In comparison to current sequential AB testing methodology, we increase statistical power by exploiting correlations between factors, enabling sequential testing and progressive early stopping, without incurring excessive false positive risk. We also demonstrate how this methodology can be extended to enable the extraction of composite global learnings from past AB tests, to accelerate future tests. We underpin our work with a solid theoretical framework that articulates the value of hierarchical estimation. We demonstrate its utility using both numerical simulations and a large set of real-world AB tests. Together, these results highlight the practical value of our approach for statistical inference in the technology industry.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑