在线控制试验分析中的偏差方差权衡
Bias Variance Tradeoff in Analysis of Online Controlled Experiments
摘要：
许多组织利用大规模在线受控实验（OCE）来加速创新。具有高统计功效以准确检测对照和治疗之间的微小差异至关重要，因为即使关键指标的微小变化也可能价值数百万美元或表明大量用户的用户不满。对于大规模OCE，持续时间通常较短（例如两周），以加快产品的更改和改进。在本文中，我们研究了两种常见的方法，用于分析在实验的时间窗口内从用户收集的使用数据，这可能在准确性和功率上有所不同。开放式方法包括所有活跃用户在整个实验期间的所有相关使用数据。有界方法包括来自用户在实验窗口中第一次变得活跃之后的每个用户的固定观察期（例如，暴露后七天）的数据。
英文摘要：
Many organizations utilize large-scale online controlled experiments (OCEs) to accelerate innovation. Having high statistical power to detect small differences between control and treatment accurately is critical, as even small changes in key metrics can be worth millions of dollars or indicate user dissatisfaction for a very large number of users. For large-scale OCE, the duration is typically short (e.g. two weeks) to expedite changes and improvements to the product. In this paper, we examine two common approaches for analyzing usage data collected from users within the time window of an experiment, which can differ in accuracy and power. The open approach includes all relevant usage data from all active users for the entire duration of the experiment. The bounded approach includes data from a fixed period of observation for each user (e.g. seven days after exposure) after the first time a user became active in the experiment window.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
