arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1412.3103cs.IR
收藏

自适应局部敏感哈希算法的序贯假设检验

Sequential Hypothesis Tests for Adaptive Locality Sensitive Hashing

Aniket Chakrabarti, Srinivasan Parthasarathy

摘要:

所有对相似性搜索是一个问题,其中一组数据对象是给定的,任务是找到所有对对象的相似性高于一定的阈值为一个给定的相似性度量的利益。当点的数量或维度很高时,标准解决方案无法优雅地缩放。近似的解决方案,如局部敏感哈希(LSH)及其贝叶斯变体(BayesLSH和BayesLSHLite)在一定程度上缓解了这个问题,并提供了比传统的基于索引的方法大幅加速。BayesLSH用于修剪候选空间和计算近似相似度,而BayesLSHLite只能修剪候选空间,但需要在原始数据上精确计算相似度。因此,在显式数据表示可用且精确相似性计算不太昂贵的情况下,BayesLSHLite可以用于积极修剪候选项并提供实质性的加速,而不会在质量上损失太多。然而,在BayesLSH变体中,质量损失更高,其中显式数据表示不可用,而只有散列草图可用,并且必须近似估计相似性。在这项工作中,我们重新审视的LSH问题,从频率设置和制定顺序测试复合假设(相似性大于或小于阈值),可以利用这样的LSH算法自适应修剪候选人积极。我们提出了一个香草序贯概率定量检验(SPRT)方法的基础上,这个想法和两个新的变种。我们将这些变体扩展到需要使用固定宽度顺序置信区间生成技术计算近似相似性的情况。

英文摘要:

All pairs similarity search is a problem where a set of data objects is given and the task is to find all pairs of objects that have similarity above a certain threshold for a given similarity measure-of-interest. When the number of points or dimensionality is high, standard solutions fail to scale gracefully. Approximate solutions such as Locality Sensitive Hashing (LSH) and its Bayesian variants (BayesLSH and BayesLSHLite) alleviate the problem to some extent and provides substantial speedup over traditional index based approaches. BayesLSH is used for pruning the candidate space and computation of approximate similarity, whereas BayesLSHLite can only prune the candidates, but similarity needs to be computed exactly on the original data. Thus where ever the explicit data representation is available and exact similarity computation is not too expensive, BayesLSHLite can be used to aggressively prune candidates and provide substantial speedup without losing too much on quality. However, the loss in quality is higher in the BayesLSH variant, where explicit data representation is not available, rather only a hash sketch is available and similarity has to be estimated approximately. In this work we revisit the LSH problem from a Frequentist setting and formulate sequential tests for composite hypothesis (similarity greater than or less than threshold) that can be leveraged by such LSH algorithms for adaptively pruning candidates aggressively. We propose a vanilla sequential probability ration test (SPRT) approach based on this idea and two novel variants. We extend these variants to the case where approximate similarity needs to be computed using fixed-width sequential confidence interval generation technique.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑