Set similarity join, as well as the corresponding indexing problem set similarity search, are fundamental primitives for managing noisy or uncertain data. For example, these primitives can be used in data cleaning to identify different representations of the same object. In many cases one can represent an object as a sparse 0-1 vector, or equivalently as the set of nonzero entries in such a vector. A set similarity join can then be used to identify those pairs that have an exceptionally large dot product (or intersection, when viewed as sets). We choose to focus on identifying vectors with large Pearson correlation, but results extend to other similarity measures. In particular, we consider the indexing problem of identifying correlated vectors in a set S of vectors sampled from {0,1}^d. Given a query vector y and a parameter alpha in (0,1), we need to search for an alpha-correlated vector x in a data structure representing the vectors of S. This kind of similarity search has been intensely studied in worst-case (non-random data) settings.
Existing theoretically well-founded methods for set similarity search are often inferior to heuristics that take advantage of skew in the data distribution, i.e., widely differing frequencies of 1s across the d dimensions. The main contribution of this paper is to analyze the set similarity problem under a random data model that reflects the kind of skewed data distributions seen in practice, allowing theoretical results much stronger than what is possible in worst-case settings. Our indexing data structure is a recursive, data-dependent partitioning of vectors inspired by recent advances in set similarity search. Previous data-dependent methods do not seem to allow us to exploit skew in item frequencies, so we believe that our work sheds further light on the power of data dependence.
Microsoft Research（微软研究院）；The University of Queensland（昆士兰大学）；Tsinghua University（清华大学）
相似性搜索（最近邻搜索）是从大型数据库中寻找与查询项距离最小的数据项的问题。已经开发了各种方法来解决这个问题，最近已经投入了大量的努力近似搜索。在本文中，我们提出了一个调查的主要解决方案之一，哈希，这已经被广泛研究，因为开拓性的工作局部敏感哈希。本文将散列算法分为两大类：局部敏感散列算法（locality sensitive hashing）和学习散列算法（learning to hash），局部敏感散列算法是根据数据分布来学习散列函数，本文从散列函数的设计、散列编码空间中的距离度量和搜索机制等方面对散列算法进行了综述。
A Generic Inverted Index Framework for Similarity Search on the GPU - Technical Report
National University of Singapore（新加坡国立大学）；Univ. of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Nanyang Technological University（南洋理工大学）；Baidu Research（百度研究院）；Tencent Inc.（腾讯公司）
FLASH: Randomized Algorithms Accelerated over CPU-GPU for Ultra-High Dimensional Similarity Search
我们提出了FLASH（\textbf{F} ast \textbf{L} SH \textbf{A} lsm for \textbf{S} imilarity search accelerated with \textbf{H} PC），一个在单机上的超高维数据集的相似性搜索系统，它不需要相似性计算，是为高性能计算平台量身定制的。通过利用LSH风格的随机化索引过程，并将其与几种原则性技术相结合，如水库采样，最新进展的一次通过minwise散列，基于计数的估计，我们降低了相似性搜索的计算和并行化成本，同时保留了良好的理论保证。我们评估FLASH上的几个真正的，高维的数据集，从不同的领域，包括文本，恶意URL，点击预测，社交网络等，我们的实验揭示了新的光与数据集具有数百万个维度的困难。当前最先进的实现要么在所呈现的规模上失败，要么比闪存慢几个数量级。FLASH能够在不到10秒的时间内从零开始计算整个垃圾邮件数据集（13亿个非零值）的近似k-NN图。使用蛮力（$n^2D$）在不到10秒的时间内在webspam数据集上计算一个完整的k-NN图，将需要至少20 teraflops。我们提供闪存的CPU和GPU实现，以实现结果的可复制性。
