广义线性模型在大数据中的应用:一种分治与重组(D&R)方法
Application of generalized linear models in big data: a divide and recombine (D&R) approach
AI总结:
本文针对广义线性模型(GLMs)的分治与重组(D&R)方法缺乏标准误验证的问题,提出合并标准误计算方法与顺序划分策略,验证其效率与全数据估计相当、结果准确。
摘要:
分治与重组(D&R)是一种专为处理大型复杂数据集设计的统计方法。它将数据集划分为多个可处理的子集,随后对每个子集独立应用分析方法以得到结果,最后合并各子集的结果,得出整个数据集的分析结果。D&R策略可用于拟合广义线性模型(GLMs),以处理规模过大、无法用常规方法分析的数据集。目前已有多种适用于不同GLMs的D&R策略,其中部分具备理论依据,但缺乏实际验证。一个显著的局限是,合并后的标准误和置信区间估计缺乏理论与实践层面的合理性支撑。本文综述了面向GLMs的D&R策略,并提出了一种确定基于D&R的估计量合并标准误的方法。除传统的数据集划分流程外,我们针对GLMs的D&R估计量提出了一种名为“顺序划分”的全新划分方法。研究表明,采用所提标准误的D&R估计量可达到与全数据估计相当的效率。我们在一个大型合成数据集上验证了该方法,证实D&R所得结果准确,且与其他现有R包的结果完全一致。
英文摘要:
D&R is a statistical approach designed to handle large and complex datasets. It partitions the dataset into several manageable subsets and subsequently applies the analytic method to each subset independently to obtain results. Finally, the results from each subset are combined to yield the results for the entire dataset. D&R strategies can be implemented to fit GLMs to datasets too large for conventional methods. Several D&R strategies are available for different GLMs, some of which are theoretically justified but lack practical validation. A significant limitation is the theoretical and practical justification for estimating combined standard errors and confidence intervals. This paper reviews D&R strategies for GLMs and proposes a method to determine the combined standard error for D&R-based estimators. In addition to the traditional dataset division procedures, we propose a different division method named sequential partitioning for D&R-based estimators on GLMs. We show that the obtained D&R estimator with the proposed standard error attains equivalent efficiency as the full data estimate. We illustrate this on a large synthetic dataset and verify that the results from D&R are accurate and identical to those from other available R packages.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。