arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1602.08393cs.DScs.IR
收藏

常数时间内的精确加权Minwise Hash

Exact Weighted Minwise Hashing in Constant Time

Anshumali Shrivastava

摘要:

加权最小散列算法(WMH)是许多著名的近似算法所需的基本子程序之一,通常用于工业实践中的大规模搜索和学习。算法的资源瓶颈是计算多个(通常是几百到几千个)独立的数据散列。最快的哈希算法是Ioffe的\cite{Proc:Ioffe_ICDM10},它需要在整个数据向量$O(d)$($d$是非零的数量)上通过一次,以计算一个哈希。然而,多个散列的要求需要对数据进行数百或数千次传递。这对于现代大规模数据集来说是非常昂贵的。在这项工作中,我们打破了这一昂贵的障碍,并显示了一个预期的常数摊销时间算法,计算$k$独立和无偏WMH的时间$O(k)$,而不是$O(dk)$所需的Ioffe的方法。此外,与现有技术方法所需的大约$64$位相比,我们的建议仅需要每个散列值的几位(5 - 9位)存储。在真实数据集上的实验评估表明,对于计算500 WMH,我们的建议可以比Ioffe的方法快60000倍,而不会损失任何准确性。我们的方法也是大约100倍的速度比近似算法利用有效的"密集"一个置换散列方案\cite{Proc:OneHashLSH_ICML14}。鉴于我们的方法的简单性及其显著的优点,我们希望它将在实践中取代现有的实现。

英文摘要:

Weighted minwise hashing (WMH) is one of the fundamental subroutine, required by many celebrated approximation algorithms, commonly adopted in industrial practice for large scale-search and learning. The resource bottleneck of the algorithms is the computation of multiple (typically a few hundreds to thousands) independent hashes of the data. The fastest hashing algorithm is by Ioffe \cite{Proc:Ioffe_ICDM10}, which requires one pass over the entire data vector, $O(d)$ ($d$ is the number of non-zeros), for computing one hash. However, the requirement of multiple hashes demands hundreds or thousands passes over the data. This is very costly for modern massive dataset. In this work, we break this expensive barrier and show an expected constant amortized time algorithm which computes $k$ independent and unbiased WMH in time $O(k)$ instead of $O(dk)$ required by Ioffe's method. Moreover, our proposal only needs a few bits (5 - 9 bits) of storage per hash value compared to around $64$ bits required by the state-of-art-methodologies. Experimental evaluations, on real datasets, show that for computing 500 WMH, our proposal can be 60000x faster than the Ioffe's method without losing any accuracy. Our method is also around 100x faster than approximate heuristics capitalizing on the efficient "densified" one permutation hashing schemes \cite{Proc:OneHashLSH_ICML14}. Given the simplicity of our approach and its significant advantages, we hope that it will replace existing implementations in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑