皮尔逊相关系数及其统计显著性对在线数据敏感性的高效算法
Efficient algorithms for the sensitivities of the Pearson correlation coefficient and its statistical significance to online data
AI总结:
本文基于Welford在线算法建立理论框架,推导出皮尔逊相关系数及其p值在新增数据下最大变化的闭式解,并给出线性与常数时间的高效算法。
摘要:
可靠地测量双变量数据的共线性在统计学中至关重要,尤其是在时间序列分析或进行中的研究中,因为新到的观测值可能显著影响当前的共线性估计。利用Welford在线样本方差算法中的恒等式,我们建立了一个严格的理论框架,用于分析额外数据可能引起的皮尔逊相关系数及其p值的最大变化。此外,我们证明了由此产生的优化问题具有简洁的闭式解,可以通过线性时间和常数时间算法精确计算。我们的工作不仅为稳健相关性度量开辟了新的理论途径,而且对计量经济学、运筹学、临床试验、气候学、差分隐私和生物信息学等学科具有广泛的实际意义。我们的算法以Cython封装的C语言实现,软件代码可在https://github.com/marc-harary/sensitivity获取,以确保可复现性、实际部署和未来的理论发展。
英文摘要:
Reliably measuring the collinearity of bivariate data is crucial in statistics, particularly for time-series analysis or ongoing studies in which incoming observations can significantly impact current collinearity estimates. Leveraging identities from Welford's online algorithm for sample variance, we develop a rigorous theoretical framework for analyzing the maximal change to the Pearson correlation coefficient and its p-value that can be induced by additional data. Further, we show that the resulting optimization problems yield elegant closed-form solutions that can be accurately computed by linear- and constant-time algorithms. Our work not only creates new theoretical avenues for robust correlation measures, but also has broad practical implications for disciplines that span econometrics, operations research, clinical trials, climatology, differential privacy, and bioinformatics. Software implementations of our algorithms in Cython-wrapped C are made available at https://github.com/marc-harary/sensitivity for reproducibility, practical deployment, and future theoretical development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。