arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.00631stat.MEecon.EMstat.ML
收藏

关于海量数据的Subbagging估计

On the Subbagging Estimation for Massive Data

Tao Zou, Xian Li, Xuan Liang, Hansheng Wang

摘要:

针对计算机内存受限的大数据分析问题,提出了一种子样本聚集估计方法。具体而言,对于大小为$N$的整个数据集,随机抽取$m_N$子样本,并且对满足内存约束的子样本大小为$k_N\ll N$的每个子样本进行均匀采样而不进行替换。聚合$m_N$子样本的估计量可以导致子装袋估计。为了分析子装袋估计的理论性质,我们采用不完全$U$ -统计理论与无限阶核,允许重叠的抽样过程中抽取的子样本。利用这一新的理论框架,我们证明了通过适当地选择$k_N$和$m_N$的超参数,在$(k_Nm_N)/N\to α\in (0,\infty]$的条件下,Subbagging估计可以达到$\sqrt{N}$ -相合性和渐近正态性.与全样本估计相比,我们从理论上证明了$\sqrt{N}$ -相容子装袋估计在其渐近方差中具有$1/α$的膨胀率。仿真实验证明了有限样本的性能。通过对美国航空公司数据集的分析,说明子装袋估计在数值上接近于全样本估计,并且在内存约束下计算速度快。

英文摘要:

This article introduces subbagging (subsample aggregating) estimation approaches for big data analysis with memory constraints of computers. Specifically, for the whole dataset with size $N$, $m_N$ subsamples are randomly drawn, and each subsample with a subsample size $k_N\ll N$ to meet the memory constraint is sampled uniformly without replacement. Aggregating the estimators of $m_N$ subsamples can lead to subbagging estimation. To analyze the theoretical properties of the subbagging estimator, we adapt the incomplete $U$-statistics theory with an infinite order kernel to allow overlapping drawn subsamples in the sampling procedure. Utilizing this novel theoretical framework, we demonstrate that via a proper hyperparameter selection of $k_N$ and $m_N$, the subbagging estimator can achieve $\sqrt{N}$-consistency and asymptotic normality under the condition $(k_Nm_N)/N\to α\in (0,\infty]$. Compared to the full sample estimator, we theoretically show that the $\sqrt{N}$-consistent subbagging estimator has an inflation rate of $1/α$ in its asymptotic variance. Simulation experiments are presented to demonstrate the finite sample performances. An American airline dataset is analyzed to illustrate that the subbagging estimate is numerically close to the full sample estimate, and can be computationally fast under the memory constraint.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑