新颖性与首要性:在线实验的长期估计
Novelty and Primacy: A Long-Term Estimator for Online Experiments
摘要:
在线实验是评估对用户体验影响和加速软件创新的黄金标准。然而,由于实验通常在持续时间上是有限的,观察到的治疗效果并不总是永久稳定的,有时会显示出随时间增加或减少的模式。治疗效果随时间变化的原因有多种。在本文中,我们专注于一个特定的原因,用户学习,这主要是与新颖性或首要性。新颖性描述了使用新技术的愿望,这种愿望往往会随着时间的推移而减弱。Primacy描述了由于采用创新而日益增长的技术参与。用户学习评估至关重要,因为它使实验对可信度负责,通过提供对预期影响的长期看法使组织能够做出更好的决策,并防止用户不满。在本文中,我们提出了一种观察方法,基于差异中的差异技术来估计用户学习的规模。我们在微软的许多实验中使用这种方法来测试和估计用户学习。我们将我们的方法与现有的实验方法进行比较,以显示其在易用性和更高的统计功效方面的优势,并讨论其在其他形式的治疗与时间相互作用中的局限性。
英文摘要:
Online experiments are the gold standard for evaluating impact on user experience and accelerating innovation in software. However, since experiments are typically limited in duration, observed treatment effects are not always permanently stable, sometimes revealing increasing or decreasing patterns over time. There are multiple causes for a treatment effect to change over time. In this paper, we focus on a particular cause, user-learning, which is primarily associated with novelty or primacy. Novelty describes the desire to use new technology that tends to diminish over time. Primacy describes the growing engagement with technology as a result of adoption of the innovation. User-learning estimation is critical because it holds experimentation responsible for trustworthiness, empowers organizations to make better decisions by providing a long-term view of expected impact, and prevents user dissatisfaction. In this paper, we propose an observational approach, based on difference-in-differences technique to estimate user-learning at scale. We use this approach to test and estimate user-learning in many experiments at Microsoft. We compare our approach with the existing experimental method to show its benefits in terms of ease of use and higher statistical power, and to discuss its limitation in presence of other forms of treatment interaction with time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。