arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.01421stat.AP
收藏

在线实验与代理服务:指南和案例研究

Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

Weitao Duan, Shan Ba, Chunzhe Zhang

摘要:

A/B测试已被各行业广泛采用,成为指导决策的黄金法则。然而,我们最终希望通过A/B测试的长期真北指标可能需要很长时间才能成熟。在这些情况下,预测长期指标的替代指标通常用于推断治疗是否有效。然而,由于替代很少完美地预测真北,基于替代度量的常规A/B测试往往具有高假阳性率,并且从测试中认为有利的治疗变体可能不是获胜的治疗变体。本文讨论了如何调整A/B测试比较,以保证实验结果的可信性。我们还提供了良好的替代指标的选择实用的指导方针。为了提供一个具体的例子,如何利用代理指标快速决策,我们提出了一个案例研究开发和评估预测确认雇用代理指标在LinkedIn就业市场。

英文摘要:

A/B tests have been widely adopted across industries as the golden rule that guides decision making. However, the long-term true north metrics we ultimately want to drive through A/B test may take a long time to mature. In these situations, a surrogate metric which predicts the long-term metric is often used instead to conclude whether the treatment is effective. However, because the surrogate rarely predicts the true north perfectly, a regular A/B test based on surrogate metrics tends to have high false positive rate and the treatment variant deemed favorable from the test may not be the winning one. In this paper, we discuss how to adjust the A/B testing comparison to ensure experiment results are trustworthy. We also provide practical guidelines on the choice of good surrogate metrics. To provide a concrete example of how to leverage surrogate metrics for fast decision making, we present a case study on developing and evaluating the predicted confirmed hire surrogate metric in LinkedIn job marketplace.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑