解决在线实验中隐藏的缺陷
Addressing Hidden Imperfections in Online Experimentation
摘要:
科技公司越来越多地使用随机对照试验(RCT)作为其开发过程的一部分。尽管对工程系统和数据仪表进行了精细的控制,但这些RCT仍然可以不完美地执行。事实上,在线实验受到许多与生物医学RCT相同的偏见的影响,包括选择和用户活动偏见,选择偏见,不遵守治疗,更普遍的是,测试感兴趣问题的能力面临挑战。这些不完善的结果可能导致估计的因果效应存在偏差,统计功效损失,效应衰减,甚至需要重新构建可以回答的问题。本文旨在让实验从业者更加了解技术行业随机对照试验中的缺陷,这些缺陷可能隐藏在整个工程堆栈或设计过程中。
英文摘要:
Technology companies are increasingly using randomized controlled trials (RCTs) as part of their development process. Despite having fine control over engineering systems and data instrumentation, these RCTs can still be imperfectly executed. In fact, online experimentation suffers from many of the same biases seen in biomedical RCTs including opt-in and user activity bias, selection bias, non-compliance with the treatment, and more generally, challenges in the ability to test the question of interest. The result of these imperfections can lead to a bias in the estimated causal effect, a loss in statistical power, an attenuation of the effect, or even a need to reframe the question that can be answered. This paper aims to make practitioners of experimentation more aware of imperfections in technology-industry RCTs, which can be hidden throughout the engineering stack or in the design process.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。