为MinHash而不是SimHash辩护
In Defense of MinHash Over SimHash
摘要：
MinHash和SimHash是两种广泛应用于大规模数据处理应用的局部敏感哈希（LSH）算法。决定哪种LSH用于手头的特定问题是一个重要的问题，在现有的文献中没有明确的答案。在这项研究中，我们提供了一个理论答案（通过实验验证），即MinHash在数据为二进制时几乎总是优于SimHash，这在搜索等实践中很常见。MinHash的冲突概率是相似性相似度（$\mathcal{R}$）的函数，而SimHash的冲突概率是余弦相似度（$\mathcal{S}$）的函数。为了提供一个共同的基础进行比较，我们评估检索结果的$\mathcal{S}$为MinHash和SimHash。该评估是有效的，因为我们可以通过使用一般不等式$\mathcal{S}^2\leq \mathcal{R}\leq \frac{\mathcal{S}}{2-\mathcal{S}}$来证明MinHash是关于$\mathcal{S}$的有效LSH。我们的最坏情况分析表明，MinHash在高相似性区域的性能明显优于SimHash。有趣的是，我们密集的实验表明，即使在大多数数据点彼此不太相似的数据集中，MinHash也比SimHash好得多。这部分是因为，在实际数据中，通常$\mathcal{R}\geq \frac{\mathcal{S}}{z-\mathcal{S}}$在$z$仅略大于2的情况下成立（例如，$z\leq 2.1$）。我们通过假设$\frac{\mathcal{S}}{z-\mathcal{S}}\leq \mathcal{R}\leq \frac{\mathcal{S}}{2-\mathcal{S}}$进行的限制性最差情况分析表明，即使在低相似性区域，MinHash也确实显著优于SimHash。我们相信本文的结果将为实际搜索提供有价值的指导，特别是当数据稀疏时。
英文摘要：
MinHash and SimHash are the two widely adopted Locality Sensitive Hashing (LSH) algorithms for large-scale data processing applications. Deciding which LSH to use for a particular problem at hand is an important question, which has no clear answer in the existing literature. In this study, we provide a theoretical answer (validated by experiments) that MinHash virtually always outperforms SimHash when the data are binary, as common in practice such as search. The collision probability of MinHash is a function of resemblance similarity ($\mathcal{R}$), while the collision probability of SimHash is a function of cosine similarity ($\mathcal{S}$). To provide a common basis for comparison, we evaluate retrieval results in terms of $\mathcal{S}$ for both MinHash and SimHash. This evaluation is valid as we can prove that MinHash is a valid LSH with respect to $\mathcal{S}$, by using a general inequality $\mathcal{S}^2\leq \mathcal{R}\leq \frac{\mathcal{S}}{2-\mathcal{S}}$. Our worst case analysis can show that MinHash significantly outperforms SimHash in high similarity region. Interestingly, our intensive experiments reveal that MinHash is also substantially better than SimHash even in datasets where most of the data points are not too similar to each other. This is partly because, in practical data, often $\mathcal{R}\geq \frac{\mathcal{S}}{z-\mathcal{S}}$ holds where $z$ is only slightly larger than 2 (e.g., $z\leq 2.1$). Our restricted worst case analysis by assuming $\frac{\mathcal{S}}{z-\mathcal{S}}\leq \mathcal{R}\leq \frac{\mathcal{S}}{2-\mathcal{S}}$ shows that MinHash indeed significantly outperforms SimHash even in low similarity region. We believe the results in this paper will provide valuable guidelines for search in practice, especially when the data are sparse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
