当主要目标不敏感或存在延迟时,实验者可能会转而关注从次要结果中得出的代理指标。例如,科技公司经常从产品干预对短期用户参与度信号的影响中推断其长期影响。我们考虑对许多历史实验进行元分析,以学习这些结果上处理效应的协方差,这可以支持此类代理指标的构建。即使实验数量充足,如果处理效应较弱,跨实验的估计处理效应协方差也可能存在高度偏差。我们通过受弱工具变量分析启发的技术克服了这一问题。我们表明,Limited Information Maximum Likelihood (LIML) 学习到的参数等价于对处理效应散点图的变换拟合全最小二乘法,而 Jackknife Instrumental Variables Estimation (JIVE) 学习到的另一个参数可从跨实验的 Jackknife 协方差矩阵的平均值中计算得出。我们还提出了在同方差条件下后一估计量的全协方差估计器,它等价于一个 $k$-class 估计器。我们展示了如何在各种结构模型下使用这些参数来构建无偏的代理指标。最后,我们讨论了我们的方法在 Netflix 的实际应用。
英文摘要:
When primary objectives are insensitive or delayed, experimenters may instead focus on proxy metrics derived from secondary outcomes. For example, technology companies often infer the long-term impacts of product interventions from their effects on short-term user engagement signals. We consider the meta-analysis of many historical experiments to learn the covariance of treatment effects on these outcomes, which can support the construction of such proxies. Even when experiments are plentiful, if treatment effects are weak, the covariance of estimated treatment effects across experiments can be highly biased. We overcome this with techniques inspired by weak instrumental variable analysis. We show that Limited Information Maximum Likelihood (LIML) learns a parameter equivalent to fitting total least squares to a transformation of the scatterplot of treatment effects, and that Jackknife Instrumental Variables Estimation (JIVE) learns another parameter computable from the average of Jackknifed covariance matrices across experiments. We also present a total covariance estimator for the latter estimand under homoskedasticity, which is equivalent to a $k$-class estimator. We show how these parameters can be used to construct unbiased proxy metrics under various structural models. Lastly, we discuss the real-world application of our methods at Netflix.
Introducing the CP-plot for Causal Inference with Observational Studies
Qiuzhen College, Tsinghua University(清华大学丘成桐学堂);Yau Mathematical Sciences Center, Tsinghua University(清华大学丘成桐数学科学中心);Department of Statistics, University of California, Berkeley(加州大学伯克利分校统计学系)
Assessing an Alternative for `Negative Variance Components': A Gentle Introduction to Bayesian Covariance Structure Modelling for Negative Associations Among Patients with Personalized Treatments