arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.05591stat.MEmath.STstat.TH
收藏

具有时间干扰的自适应试验设计:一种极大似然方法

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

Peter Glynn, Ramesh Johari, Mohammad Rasouli

摘要:

假设在线平台想要比较治疗和控制策略,例如,在共乘系统中使用两种不同的匹配算法,或者在在线零售站点中使用两种不同的库存管理算法。标准的随机对照试验通常是不可行的,因为其目标是评估整个系统的政策绩效。相反,典型的当前实践涉及在固定时间长度内在两种策略之间动态地交替,并且比较在它们运行的时间间隔内每种策略的平均性能作为处理效应的估计。然而,这种方法受到 * 时间干扰 * 的影响:一种算法改变了第二种算法所看到的系统状态,使处理效应的估计值产生偏差。此外,这种设计的简单非自适应性质意味着它们不是样本有效的。我们开发了一个基准的理论模型,在其中研究最佳的试验设计,这种设置。我们将测试这两个策略视为估计两个未知马尔可夫链之间的稳态奖励差异的问题(即,政策)。我们假设每个链的稳态奖励的估计通过非参数极大似然进行,并搜索一致的(即,渐近无偏的)有效的试验设计(即,渐近最小方差)。这种设计的特点是相当于一个马尔可夫决策问题的最小方差目标,这样的问题一般不承认听话的解决方案。值得注意的是,在我们的设置中,使用经典的鞅分析的马尔可夫链通过泊松方程的一个新的应用,我们通过一个简洁的凸优化问题的有效设计的特征。我们使用这个特性,提出了一个一致的,有效的在线试验设计,自适应采样的两个马尔可夫链。

英文摘要:

Suppose an online platform wants to compare a treatment and control policy, e.g., two different matching algorithms in a ridesharing system, or two different inventory management algorithms in an online retail site. Standard randomized controlled trials are typically not feasible, since the goal is to estimate policy performance on the entire system. Instead, the typical current practice involves dynamically alternating between the two policies for fixed lengths of time, and comparing the average performance of each over the intervals in which they were run as an estimate of the treatment effect. However, this approach suffers from *temporal interference*: one algorithm alters the state of the system as seen by the second algorithm, biasing estimates of the treatment effect. Further, the simple non-adaptive nature of such designs implies they are not sample efficient. We develop a benchmark theoretical model in which to study optimal experimental design for this setting. We view testing the two policies as the problem of estimating the steady state difference in reward between two unknown Markov chains (i.e., policies). We assume estimation of the steady state reward for each chain proceeds via nonparametric maximum likelihood, and search for consistent (i.e., asymptotically unbiased) experimental designs that are efficient (i.e., asymptotically minimum variance). Characterizing such designs is equivalent to a Markov decision problem with a minimum variance objective; such problems generally do not admit tractable solutions. Remarkably, in our setting, using a novel application of classical martingale analysis of Markov chains via Poisson's equation, we characterize efficient designs via a succinct convex optimization problem. We use this characterization to propose a consistent, efficient online experimental design that adaptively samples the two Markov chains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑