与维数无关的相似计算
Dimension Independent Similarity Computation
摘要:
我们提出了一套算法的维数无关的相似性计算(DISCO),以计算非常高维的稀疏向量之间的所有成对的相似性。我们所有的结果都可以证明是与维度无关的,这意味着除了微不足道地读取数据的初始成本之外,所有后续操作都与维度无关,因此维度可以非常大。我们研究余弦,骰子,重叠,和Jaccard相似性措施。对于Jaccard相似性,我们包括MinHash的改进版本。我们的结果是面向MapReduce框架。我们经验验证我们的定理在大规模使用的数据从社交网站Twitter。在撰写本文时,我们的算法已经在twitter.com上线。
英文摘要:
We present a suite of algorithms for Dimension Independent Similarity Computation (DISCO) to compute all pairwise similarities between very high dimensional sparse vectors. All of our results are provably independent of dimension, meaning apart from the initial cost of trivially reading in the data, all subsequent operations are independent of the dimension, thus the dimension can be very large. We study Cosine, Dice, Overlap, and the Jaccard similarity measures. For Jaccard similiarity we include an improved version of MinHash. Our results are geared toward the MapReduce framework. We empirically validate our theorems at large scale using data from the social networking site Twitter. At time of writing, our algorithms are live in production at twitter.com.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。