针对分布式数据流中heavy distinct hitters问题，提出结合sketch与采样技术的Sampling Space-Saving Set Sketch算法，在大小、速度、精度、可合并性和可逆性上均具优势且实验证明其优越性。
摘要：
大型分布式数据流如今已无处不在。具有低内存开销的高精度 sketch 已成为分析此类数据的事实标准方法。例如，如果我们希望按某个标签对数据进行分组，并在固定内存下报告最大的计数值，我们需要借助可合并的 heavy hitter sketch，它能提供高精度的近似计数。类似地，如果我们希望在固定内存下跟踪分布在多个流中的单个集合内不同元素的数量，我们可以借助可合并的 count distinct sketch，它能提供高精度的集合基数。\n 如果我们试图跟踪多个集合的基数并仅报告其中最大的集合，为每个集合维护单独的 count distinct sketch 会变得难以管理，尤其是在集合数量事先未知的情况下。我们考虑了 heavy hitters 问题与 count distinct 问题的自然结合，即 heavy distinct hitters 问题：给定一个由 $(\ell, x)$ 对组成的流，在仅使用常数内存的情况下，找到所有与大量不同元素 $x$ 配对的标签 $\ell$。\n 以往关于 heavy distinct hitters 的工作均未能在大型分布式数据流场景中实现实用。我们提出了一种新算法，即 Sampling Space-Saving Set Sketch，它结合了 sketch 与采样技术，并在大小、速度、精度、可合并性和可逆性方面具备所有期望的特性。我们将该算法与 heavy distinct hitters 问题的几种现有解决方案进行了比较，并在多个数据集上提供了实验结果，证明了新 sketch 的优越性。
英文摘要：
Large, distributed data streams are now ubiquitous. High-accuracy sketches with low memory overhead have become the de facto method for analyzing this data. For instance, if we wish to group data by some label and report the largest counts using fixed memory, we need to turn to mergeable heavy hitter sketches that can provide highly accurate approximate counts. Similarly, if we wish to keep track of the number of distinct items in a single set spread across several streams using fixed memory, we can turn to mergeable count distinct sketches that can provide highly accurate set cardinalities.
If we were to try to keep track of the cardinality of multiple sets and report only on the largest ones, maintaining individual count distinct sketches for each set can grow unwieldy, especially if the number of sets is not known in advance. We consider the natural combination of the heavy hitters problem with the count distinct problem, the heavy distinct hitters problem: given a stream of $(\ell, x)$ pairs, find all the labels $\ell$ that are paired with a large number of distinct items $x$ using only constant memory.
No previous work on heavy distinct hitters has managed to be of practical use in the large, distributed data stream setting. We propose a new algorithm, the Sampling Space-Saving Set Sketch, which combines sketching and sampling techniques and has all the desired properties for size, speed, accuracy, mergeability, and invertibility. We compare our algorithm to several existing solutions to the heavy distinct hitters problem, and provide experimental results across several data sets showing the superiority of the new sketch.
Composable Sketches for Functions of Frequencies: Beyond the Worst Case
最近，人们越来越关注使用机器学习技术来改进经典算法。在本文中，我们研究当它是可能的构造紧凑的，可组合的草图加权抽样和统计估计根据数据频率的函数。这种结构现在是大规模数据分析和机器学习管道的核心组件。然而，许多常见的功能，如阈值和p > 2的p阶频率矩，在最坏的情况下需要多项式大小的草图。我们探讨了两种不同类型的假设下的最坏情况下的性能。第一个是获得关于项目频率的嘈杂建议。这延续了Hsu et al.（ICLR 2019）的工作路线，他们假设预测是由机器学习模型提供的。第二个是在有限的一类输入频率分布上提供有保证的性能，这些分布与实践中观察到的情况更好地一致。这扩展了Charikar et al.（ICALP 2002）的开创性论文中关于Zipfian分布下的重打击者的工作。令人惊讶的是，我们的分析和经验表明，“在实践中”小多面体大小的草图提供准确的“硬”功能。
Online Rack Placement in Large-Scale Data Centers: Online Sampling Optimization and Deployment
Microsoft（微软）；Sloan School of Management, Operations Research Center, Massachusetts Institute of Technology（麻省理工学院斯隆管理学院运筹学中心）；Microsoft Research（微软研究院）
