arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.00650cs.DS
收藏

数据流频率估计的双哈希算法

Double-Hashing Algorithm for Frequency Estimation in Data Streams

Nikita Seleznev, Senthil Kumar, C. Bayan Bruss

摘要:

元素的频率估计是数据流分析和机器学习应用的重要任务。该问题通常通过使用具有次线性空间数据结构的流算法来解决。这些算法允许在使用有限的数据存储的同时处理大数据。常用的流式算法,如count-min sketch,有很多优点,但没有考虑到性能优化的数据流的属性。在本文中,我们介绍了一种新的双哈希算法,提供了灵活性,优化流算法取决于给定流的属性。在双散列方法中,首先采用标准流算法来获得元素频率的估计。该估计值是使用流的一部分导出的,并且允许识别重打击者。接下来,它使用修改的哈希表,其中重命中者被映射到各个桶中,而其他流元素被映射到剩余的桶中。最后,基于所构造的哈希表在整个数据流上利用任何流算法来估计元素频率。我们证明了合成数据和互联网查询日志数据集,我们的方法是能够提高频率估计,由于从哈希过程中删除重打击者,从而减少冲突的哈希表。我们的方法避免了采用额外的机器学习模型来识别重量级人物,从而降低了算法复杂度并简化了实现。此外,因为它不依赖于流元素的特定特征来识别重打击者,所以它适用于各种各样的流。此外,我们提出了一个程序,如何动态调整建议的双哈希算法时,流中的元素的频率随时间而变化。

英文摘要:

Frequency estimation of elements is an important task for summarizing data streams and machine learning applications. The problem is often addressed by using streaming algorithms with sublinear space data structures. These algorithms allow processing of large data while using limited data storage. Commonly used streaming algorithms, such as count-min sketch, have many advantages, but do not take into account properties of a data stream for performance optimization. In the present paper we introduce a novel double-hashing algorithm that provides flexibility to optimize streaming algorithms depending on the properties of a given stream. In the double-hashing approach, first a standard streaming algorithm is employed to obtain an estimate of the element frequencies. This estimate is derived using a fraction of the stream and allows identification of the heavy hitters. Next, it uses a modified hash table where the heavy hitters are mapped into individual buckets and other stream elements are mapped into the remaining buckets. Finally, the element frequencies are estimated based on the constructed hash table over the entire data stream with any streaming algorithm. We demonstrate on both synthetic data and an internet query log dataset that our approach is capable of improving frequency estimation due to removing heavy hitters from the hashing process and, thus, reducing collisions in the hash table. Our approach avoids employing additional machine learning models to identify heavy hitters and, thus, reduces algorithm complexity and streamlines implementation. Moreover, because it is not dependent on specific features of the stream elements for identifying heavy hitters, it is applicable to a large variety of streams. In addition, we propose a procedure on how to dynamically adjust the proposed double-hashing algorithm when frequencies of the elements in a stream are changing over time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑