From scientific experiments to online A/B testing, the previously observed data often affects how future experiments are performed, which in turn affects which data will be collected. Such adaptivity introduces complex correlations between the data and the collection procedure. In this paper, we prove that when the data collection procedure satisfies natural conditions, then sample means of the data have systematic \emph{negative} biases. As an example, consider an adaptive clinical trial where additional data points are more likely to be tested for treatments that show initial promise. Our surprising result implies that the average observed treatment effects would underestimate the true effects of each treatment. We quantitatively analyze the magnitude and behavior of this negative bias in a variety of settings. We also propose a novel debiasing algorithm based on selective inference techniques. In experiments, our method can effectively reduce bias and estimation error.
Challenges in Statistical Analysis of Data Collected by a Bandit Algorithm: An Empirical Exploration in Applications to Adaptively Randomized Experiments
University of Toronto(多伦多大学);Sapienza University of Rome(罗马第一大学);University of Cambridge(剑桥大学);Universitè Laval(拉瓦尔大学);Carleton College(卡尔顿学院)
几十年来,多臂强盗算法一直被认为是自适应随机实验的有用方法。在此类实验中,算法会改变分配给参与者的手臂(例如帮助学生学习的替代干预措施),目标是为尽可能多的参与者分配更高回报的手臂。我们应用强盗算法汤普森采样(TS)运行自适应实验在三个大学类。教师看到了尝试快速使用数据给他们的学生在实验中更好的武器(例如,更好地解释一个概念)的巨大价值。然而,我们的部署说明了科学家和从业者使用这种自适应实验的主要障碍:与传统的均匀随机分配相比,缺乏对特定现实世界实验的统计分析受到影响程度的量化见解(Pallmann et al,2018; FDA,2019)。因此,我们使用我们的案例研究的无处不在的双臂二进制奖励设置经验调查的影响,使用汤普森采样,而不是均匀的随机分配。在这种情况下,使用常见的统计假设检验,我们表明,使用TS收集数据可以使假阳性率(FPR;当不存在差异时错误地报告差异)和假阴性率(FNR;当存在差异时未能报告差异)增加一倍。
Days Alive at Home vs Out of Hospital: Why the Difference Matters for Trial Endpoints
MRC Biostatistics Unit, University of Cambridge(剑桥大学医学研究委员会生物统计单元);Papworth Trials Unit Collaboration, Royal Papworth Hospital(皇家帕普沃斯医院帕沃斯试验单位合作组织);University of Cambridge(剑桥大学);Department of Anaesthesia, Royal Papworth Hospital(皇家帕普沃斯医院麻醉科)