arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.09261cs.DScs.DBcs.LG
收藏

数据流中的频率估计:学习最佳散列方案

Frequency Estimation in Data Streams: Learning the Optimal Hashing Scheme

Massachusetts Institute of Technology(麻省理工学院)

Dimitris Bertsimas, Vassilis Digalakis

摘要:

我们提出了一种新的方法,基于优化和机器学习的数据流中的频率估计的问题。相反,国家的最先进的流频率估计算法,这在很大程度上依赖于随机散列使用有限的存储来维持数据流的频率分布,所提出的方法利用观察到的流前缀,以接近最佳的散列元素和压缩的目标频率分布。我们开发了一个精确的混合整数线性优化公式,这使我们能够计算最佳或接近最佳的散列方案中看到的元素所观察到的流前缀,然后,我们使用机器学习散列看不见的元素。此外,我们开发了一个有效的块坐标下降算法,它,我们的经验表明,产生高质量的解决方案,并在一个特殊的情况下,我们能够解决所提出的制定正是在线性时间使用动态规划。我们实证评估所提出的方法在合成数据集和现实世界的搜索查询数据。我们表明,所提出的方法优于现有的方法,其平均(每个元素)的估计误差和45-90%的估计误差的预期幅度方面的一至两个数量级。

英文摘要:

We present a novel approach for the problem of frequency estimation in data streams that is based on optimization and machine learning. Contrary to state-of-the-art streaming frequency estimation algorithms, which heavily rely on random hashing to maintain the frequency distribution of the data steam using limited storage, the proposed approach exploits an observed stream prefix to near-optimally hash elements and compress the target frequency distribution. We develop an exact mixed-integer linear optimization formulation, which enables us to compute optimal or near-optimal hashing schemes for elements seen in the observed stream prefix; then, we use machine learning to hash unseen elements. Further, we develop an efficient block coordinate descent algorithm, which, as we empirically show, produces high quality solutions, and, in a special case, we are able to solve the proposed formulation exactly in linear time using dynamic programming. We empirically evaluate the proposed approach both on synthetic datasets and on real-world search query data. We show that the proposed approach outperforms existing approaches by one to two orders of magnitude in terms of its average (per element) estimation error and by 45-90% in terms of its expected magnitude of estimation error.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑