非聚集子集和与频繁项估计的数据草图
Data Sketches for Disaggregated Subset Sum and Frequent Item Estimation
摘要：
介绍并研究了一种新的用于处理海量数据集的数据草图。它解决了两个常见的问题：1）计算任意过滤条件下的总和，2）识别数据集中的频繁项或重命中项。对于前者，该草图提供了具有最先进精度的无偏估计。当数据被分解时，它可以处理具有挑战性的场景，因此计算每单位的感兴趣指标需要昂贵的聚合。例如，感兴趣的度量可以是每个用户的总点击，而原始数据是每个用户具有多行的点击流。因此，该草图适用于广泛的应用，包括计算广告预测的历史点击率，报告来自事件流的用户指标，以及测量IP流的网络流量。我们证明和经验表明，草图有良好的性能，分解子集和估计和频繁项目问题。在身份证上。数据，它不仅挑选出频繁项目，但给出强相合估计的比例，每个频繁项目。由此产生的草图渐近地绘制出与样本大小成比例的概率，该概率对于估计数据的总和是最佳的。对于非I.I.D.数据，我们表明，它通常比随机抽样的频繁项问题，从来没有做得更糟。对于子集和估计，我们表明，即使是病理序列，方差接近最优抽样设计。从经验上讲，尽管在分类数据上操作的缺点，我们的方法匹配或优于优先级采样，这是一种用于预聚合数据的最先进的方法，并且与均匀采样相比，在偏斜数据上的表现更好。我们提出了扩展的草图，允许它被用于多个数据集相结合，在分布式系统中，并为时间衰减聚合。
英文摘要：
We introduce and study a new data sketch for processing massive datasets. It addresses two common problems: 1) computing a sum given arbitrary filter conditions and 2) identifying the frequent items or heavy hitters in a data set. For the former, the sketch provides unbiased estimates with state of the art accuracy. It handles the challenging scenario when the data is disaggregated so that computing the per unit metric of interest requires an expensive aggregation. For example, the metric of interest may be total clicks per user while the raw data is a click stream with multiple rows per user. Thus the sketch is suitable for use in a wide range of applications including computing historical click through rates for ad prediction, reporting user metrics from event streams, and measuring network traffic for IP flows. We prove and empirically show the sketch has good properties for both the disaggregated subset sum estimation and frequent item problems. On i.i.d. data, it not only picks out the frequent items but gives strongly consistent estimates for the proportion of each frequent item. The resulting sketch asymptotically draws a probability proportional to size sample that is optimal for estimating sums over the data. For non i.i.d. data, we show that it typically does much better than random sampling for the frequent item problem and never does worse. For subset sum estimation, we show that even for pathological sequences, the variance is close to that of an optimal sampling design. Empirically, despite the disadvantage of operating on disaggregated data, our method matches or bests priority sampling, a state of the art method for pre-aggregated data and performs orders of magnitude better on skewed data compared to uniform sampling. We propose extensions to the sketch that allow it to be used in combining multiple data sets, in distributed systems, and for time decayed aggregation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
