AI 中文总结
针对大规模在线实验含噪声效应估计的问题，提出基于SIMEX的非参数方法，可从已知方差的估计平均处理效应中估计真实效应分布，性能接近参数经验贝叶斯方法，为表征效应量分布提供灵活途径。
AI 中文摘要
大规模在线实验会产生含噪声的效应估计值，这可能夸大增益并使关于上线和测试政策的决策复杂化。我们提出一种基于SIMulation-EXtrapolation（SIMEX，模拟外推）的非参数方法，用于从具有已知方差的估计平均处理效应中估计真实效应的潜在分布。该方法在逐步添加更多模拟测量噪声后评估分位数，并在强制分位数单调性的同时，将所得的逆累积分布函数外推至零噪声设置。在具有正态分布真实效应和测量误差的合成示例中，该方法能恢复潜在的效应分布，且性能几乎与参数经验贝叶斯正态均值方法相当。这为无需完全指定参数模型即可表征效应量分布提供了一种灵活方式。
英文摘要
Large-scale online experimentation produces noisy effect estimates, which can overstate gains and complicate decisions about launches and testing policies. We propose a nonparametric method based on SIMulation-EXtrapolation (SIMEX) to estimate the latent distribution of true effects from estimated average treatment effects with known variances. The method evaluates quantiles after adding progressively more simulated measurement noise and extrapolates the resulting inverse cumulative distribution function to the zero-noise setting while enforcing monotonicity of the quantiles. In a synthetic example with normally distributed true effects and measurement error, the method recovers the underlying effect distribution and performs nearly as well as a parametric empirical Bayes normal means approach. This provides a flexible way to characterize effect-size distributions without fully specifying a parametric model.
Journal refConference on Digital Experimentation @ MIT (CODE@MIT'24), Cambridge, MA, 2024
