arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.09027stat.MEcs.LGecon.EMstat.AP
收藏

方差缩减结合预实验和实验证据数据

Variance reduction combining pre-experiment and in-experiment data

University of California, Berkeley(加州大学伯克利分校);Etsy, Inc.(Etsy公司)

Zhexiao Lin, Pablo Crespo

AI总结:

本文提出一种结合预实验和实验证据数据的方差缩减框架,提升A/B测试的敏感度,通过理论分析和实验证实其有效性。

摘要:

在线受控实验(A/B测试)是许多公司数据驱动决策的基础。在固定样本量约束下提高实验灵敏度需要减少平均处理效应(ATE)估计量的方差。现有的方差缩减技术如CUPED和CUPAC使用预实验数据,但其效果取决于这些数据对实验期间测量结果的预测能力。实验证据数据通常与结果更紧密相关,但使用任意后处理变量可能引入偏差。本文提出一种通用、稳健且可扩展的框架,结合预实验和实验证据数据以实现方差缩减。该框架简单、可解释且计算高效,适用于实际部署。我们开发了所提估计量的渐近理论,并提供了一致的方差估计器。在Etsy多个在线实验中的实证结果表明,即使仅纳入少量后处理协变量,也实现了显著的额外方差缩减。这些发现凸显了我们框架在提高实验灵敏度和加速数据驱动决策方面的有效性。

英文摘要:

Online controlled experiments (A/B testing) are fundamental to data-driven decision-making in many companies. Improving the sensitivity of these experiments under fixed sample size constraints requires reducing the variance of the average treatment effect (ATE) estimator. Existing variance reduction techniques such as CUPED and CUPAC use pre-experiment data, but their effectiveness depends on how predictive those data are for outcomes measured during the experiment. In-experiment data are often more strongly correlated with the outcome, but using arbitrary post-treatment variables can introduce bias. In this paper, we propose a general, robust, and scalable framework that combines both pre-experiment and in-experiment data to achieve variance reduction. Our framework is simple, interpretable, and computationally efficient, making it practical for real-world deployment. We develop the asymptotic theory of the proposed estimator and provide consistent variance estimators. Empirical results from multiple online experiments conducted at Etsy demonstrate substantial additional variance reduction over current pipeline, even when incorporating only a few post-treatment covariates. These findings underscore the effectiveness of our framework in improving experimental sensitivity and accelerating data-driven decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑