arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1903.08762stat.AP
收藏

使用分位数排序的大规模在线实验

Large-Scale Online Experimentation with Quantile Metrics

Min Liu, Xiaohui Sun, Maneesh Varshney, Ya Xu

摘要:

在线实验(或A/B测试)已被工业界广泛采用,作为衡量产品影响的黄金标准。尽管A/B测试被广泛采用,但很少有文献讨论分位数度量的A/B测试。分位数指标,如第90百分位页面加载时间,是至关重要的A/B测试,因为许多关键的性能指标,包括网站速度和服务延迟被定义为分位数。然而,由于LinkedIn的数据大小,分位数度量A/B测试极具挑战性,因为对于相关样本的分位数没有统计上有效且可扩展的方差估计:自助估计在统计上有效,但需要数天计算;标准渐近方差估计是可扩展的,但会导致数量级低估。在本文中,我们提出了一个统计上有效的和可扩展的方法,A/B测试的分位数,是完全可推广到其他A/B测试平台。与bootstrap相比,它实现了超过500倍的速度,并且只有$2\%$机会与bootstrap估计值不同。除了方法之外,我们还分享了使用这种方法实现数据管道的方法以及对管道优化的见解。

英文摘要:

Online experimentation (or A/B testing) has been widely adopted in industry as the gold standard for measuring product impacts. Despite the wide adoption, few literatures discuss A/B testing with quantile metrics. Quantile metrics, such as 90th percentile page load time, are crucial to A/B testing as many key performance metrics including site speed and service latency are defined as quantiles. However, with LinkedIn's data size, quantile metric A/B testing is extremely challenging because there is no statistically valid and scalable variance estimator for the quantile of dependent samples: the bootstrap estimator is statistically valid, but takes days to compute; the standard asymptotic variance estimate is scalable but results in order-of-magnitude underestimation. In this paper, we present a statistically valid and scalable methodology for A/B testing with quantiles that is fully generalizable to other A/B testing platforms. It achieves over 500 times speed up compared to bootstrap and has only $2\%$ chance to differ from bootstrap estimates. Beyond methodology, we also share the implementation of a data pipeline using this methodology and insights on pipeline optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑