持续在线实验
Always-On Experimentation
- University of California, Berkeley（加州大学伯克利分校）
- Adobe Research（奥多比研究院）
- Inria（法国国家信息与自动化研究所）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
本文形式化“持续在线”实验环境，提出基于以赌注检验框架的序贯检验方法，在任意停止时间下控制错误发现率，实现时间均匀的第一类错误控制。
中文摘要 AI 辅助
生成式AI极大地加速了新治疗方法（从新型药物到在线营销活动）的构思与部署速度。因此，现代实验平台往往持续运行，治疗方法一旦就绪便加入，表现不佳时则被移除。我们将这种“持续在线”实验环境正式化，在该环境中，治疗方法可以动态生成、加入或从正在进行的实验中移除，并研究在控制错误发现率的同时决定接受或拒绝每种治疗方法的统计问题。我们开发了序贯检验方法，在任意停止时间和“可预测”的治疗安排下实现时间均匀的第一类错误控制。我们的方法基于“以赌注检验”框架：我们为检验每种治疗方法的平均治疗效果构建检验超鞅，并证明这些检验超鞅的构建在几乎必然意义下具有增长率最优性。
英文摘要
Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.
