基于桶重用的排他和非排他在线随机实验的统计性质
Statistical Properties of Exclusive and Non-exclusive Online Randomized Experiments using Bucket Reuse
摘要:
随机实验是科技行业产品开发的关键部分。经常需要运行排他性实验程序,即,不能在同一时间内在同一设备上运行的实验。这些程序意味着对随机抽样的限制,因为当前实验中的单元不能被抽样到新的单元中。此外,为了在技术上实现与大群体的这种类型的协调,群体中的单元通常被分组为“桶”,然后在桶级上进行采样。本文研究了限制抽样和桶级抽样的一些统计含义。本文的贡献是三方面的:首先,桶抽样是连接到现有的文献中的随机实验复杂的抽样设计,使建立属性的差异,平均估计的平均处理效应。在桶的随机抽样下,需要这些性质来推断总体。其次,推导了排他性实验方案所规定的抽样限制所引起的偏差。最后,模拟结果支持的理论研究结果一起提出的建议,就如何实证评估和处理这种偏见。
英文摘要:
Randomized experiments is a key part of product development in the tech industry. It is often necessary to run programs of exclusive experiments, i.e., experiments that cannot be run on the same units during the same time. These programs implies restriction on the random sampling, as units that are currently in an experiment cannot be sampled into a new one. Moreover, to technically enable this type of coordination with large populations, the units in the population are often grouped into 'buckets' and sampling is then performed on the bucket level. This paper investigates some statistical implications of both the restricted sampling and the bucket-level sampling. The contribution of this paper is threefold: First, bucket sampling is connected to the existing literature on randomized experiments in complex sampling designs which enables establishing properties of the difference-in-means estimator of the average treatment effect. These properties are needed for inference to the population under random sampling of buckets. Second, the bias introduced by restricting the sampling as imposed by programs of exclusive experiments, is derived. Finally, simulation results supporting the theoretical findings are presented together with recommendations on how to empirically evaluate and handle this bias.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。