结构化缺失的完整刻画
A Complete Characterisation of Structured Missingness
AI总结:
本文针对缺失值具有潜在结构的问题,提出一种允许缺失指示向量相互依赖的结构化缺失分类法,并将其嵌入MCAR、MAR、MNAR框架,通过模拟和临床基因组数据库实例展示其对推断与预测的影响。
摘要:
我们处理大型复杂数据源的能力不断增强,这为我们带来了新的重要应用研究问题,例如如何处理大规模数据库中的缺失值。Mitra等人(2023)指出了结构化缺失(SM)现象,即缺失性具有潜在结构。现有的缺失机制定义分类法通常假设变量的缺失指示向量 $M_1$, $M_2$, ..., $M_p$ 在条件于数据矩阵 $\mathbf{X}$ 的相关部分后是相互独立的。由于这一假设通常不适合刻画多变量环境下的结构化缺失,我们引入了一种面向结构化缺失的分类法,其中每个 ${M}_j$ 除了可以依赖于 $\mathbf{X}$ 之外,还可以依赖于 $\mathbf{M}_{-j}$(即除 ${M}_j$ 之外的所有缺失指示向量)。我们将这一新框架嵌入到已有的将缺失机制分解为MCAR、MAR和MNAR的经典体系(Rubin, 1976)中,从而能够将缺失机制重新置于更广泛的设定下,考虑 $\mathbf{X}$ 和 $\mathbf{M}_{-j}$ 对 ${M}_j$ 的联合影响。我们还通过模拟展示了结构化缺失对推断和预测的影响,并考察了在一个去标识化的美国全国性临床基因组数据库(CGDB)中出现的结构化缺失实例。我们希望激发对结构化缺失的研究兴趣,并鼓励对这一现象开展及时研究。
英文摘要:
Our capacity to process large complex data sources is ever-increasing, providing us with new, important applied research questions to address, such as how to handle missing values in large-scale databases. Mitra et al. (2023) noted the phenomenon of Structured Missingness (SM), which is where missingness has an underlying structure. Existing taxonomies for defining missingness mechanisms typically assume that variables' missingness indicator vectors $M_1$, $M_2$, ..., $M_p$ are independent after conditioning on the relevant portion of the data matrix $\mathbf{X}$. As this is often unsuitable for characterising SM in multivariate settings, we introduce a taxonomy for SM, where each ${M}_j$ can depend on $\mathbf{M}_{-j}$ (i.e., all missingness indicator vectors except ${M}_j$), in addition to $\mathbf{X}$. We embed this new framework within the well-established decomposition of mechanisms into MCAR, MAR, and MNAR (Rubin, 1976), allowing us to recast mechanisms into a broader setting, where we can consider the combined effect of $\mathbf{X}$ and $\mathbf{M}_{-j}$ on ${M}_j$. We also demonstrate, via simulations, the impact of SM on inference and prediction, and consider contextual instances of SM arising in a de-identified nationwide (US-based) clinico-genomic database (CGDB). We hope to stimulate interest in SM, and encourage timely research into this phenomenon.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。