arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1203.4903cs.DScs.DBmath.STstat.TH
收藏

从采样数据的距离估计:准确和有效

Distance Queries from Sampled Data: Accurate and Efficient

Edith Cohen

摘要:

距离查询是数据分析的基本工具。它们用于检测和定位变化,以进行异常检测、监控或规划。当重复收集测量、系统快照、内容、流量矩阵和活动日志等数据集时,距离查询特别有用。随机采样可以有效地对流式或分布式数据执行,是可扩展数据分析的重要工具。该示例构成了一个极其灵活的摘要,它自然支持域查询和统计信息的可伸缩估计,这些可以在生成示例后指定。然而,样本作为摘要的有效性取决于我们所拥有的估计量。我们推导出新的估计估计$L_p$距离采样数据。我们的估计量适用于最常见的加权抽样方案:泊松概率与大小成比例(PPS)及其固定样本量变体。它们也适用于不同数据集的样本是独立或协调的情况。我们的估计是可接受的(帕累托最优方差),并具有令人信服的属性。我们研究了我们的曼哈顿和欧几里得距离($p=1,2$)估计器在不同数据集上的性能,即使在对一小部分数据进行采样时,也能证明其可扩展性和准确性。我们的工作,第一次,有利于有效的距离估计采样数据。

英文摘要:

Distance queries are a basic tool in data analysis. They are used for detection and localization of change for the purpose of anomaly detection, monitoring, or planning. Distance queries are particularly useful when data sets such as measurements, snapshots of a system, content, traffic matrices, and activity logs are collected repeatedly. Random sampling, which can be efficiently performed over streamed or distributed data, is an important tool for scalable data analysis. The sample constitutes an extremely flexible summary, which naturally supports domain queries and scalable estimation of statistics, which can be specified after the sample is generated. The effectiveness of a sample as a summary, however, hinges on the estimators we have. We derive novel estimators for estimating $L_p$ distance from sampled data. Our estimators apply with the most common weighted sampling schemes: Poisson Probability Proportional to Size (PPS) and its fixed sample size variants. They also apply when the samples of different data sets are independent or coordinated. Our estimators are admissible (Pareto optimal in terms of variance) and have compelling properties. We study the performance of our Manhattan and Euclidean distance ($p=1,2$) estimators on diverse datasets, demonstrating scalability and accuracy even when a small fraction of the data is sampled. Our work, for the first time, facilitates effective distance estimation over sampled data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑