We formulate and optimally solve a new generalized Set Similarity Search problem, which assumes the size of the database and query sets are known in advance. By creating polylog copies of our data-structure, we optimally solve any symmetric Approximate Set Similarity Search problem, including approximate versions of Subset Search, Maximum Inner Product Search (MIPS), Jaccard Similarity Search and Partial Match.
Our algorithm can be seen as a natural generalization of previous work on Set as well as Euclidean Similarity Search, but conceptually it differs by optimally exploiting the information present in the sets as well as their complements, and doing so asymmetrically between queries and stored sets. Doing so we improve upon the best previous work: MinHash [J. Discrete Algorithms 1998], SimHash [STOC 2002], Spherical LSF [SODA 2016, 2017] and Chosen Path [STOC 2017] by as much as a factor $n^{0.14}$ in both time and space; or in the near-constant time regime, in space, by an arbitrarily large polynomial factor.
Turning the geometric concept, based on Boolean supermajority functions, into a practical algorithm requires ideas from branching random walks on $\mathbb Z^2$, for which we give the first non-asymptotic near tight analysis.
Our lower bounds follow from new hypercontractive arguments, which can be seen as characterizing the exact family of similarity search problems for which supermajorities are optimal. The optimality holds for among all hashing based data structures in the random setting, and by reductions, for 1 cell and 2 cell probe data structures. As a side effect, we obtain new hypercontractive bounds on the directed noise operator $T^{p_1 \to p_2}_ρ$.
Microsoft Research（微软研究院）；The University of Queensland（昆士兰大学）；Tsinghua University（清华大学）
相似性搜索（最近邻搜索）是从大型数据库中寻找与查询项距离最小的数据项的问题。已经开发了各种方法来解决这个问题，最近已经投入了大量的努力近似搜索。在本文中，我们提出了一个调查的主要解决方案之一，哈希，这已经被广泛研究，因为开拓性的工作局部敏感哈希。本文将散列算法分为两大类：局部敏感散列算法（locality sensitive hashing）和学习散列算法（learning to hash），局部敏感散列算法是根据数据分布来学习散列函数，本文从散列函数的设计、散列编码空间中的距离度量和搜索机制等方面对散列算法进行了综述。
This paper studies the $r$-range search problem for curves under the continuous Fréchet distance: given a dataset $S$ of $n$ polygonal curves and a threshold $r>0$, construct a data structure that, for any query curve $q$, efficiently returns all entries in $S$ with distance at most $r$ from $q$. We propose FRESH, an approximate and randomized approach for $r$-range search, that leverages on a locality sensitive hashing scheme for detecting candidate near neighbors of the query curve, and on a subsequent pruning step based on a cascade of curve simplifications. We experimentally compare \fresh…
We propose a new class of data-independent locality-sensitive hashing (LSH) algorithms based on the fruit fly olfactory circuit. The fundamental difference of this approach is that, instead of assigning hashes as dense points in a low dimensional space, hashes are assigned in a high dimensional space, which enhances their separability. We show theoretically and empirically that this new family of hash functions is locality-sensitive and preserves rank similarity for inputs in any `p space. We then analyze different variations on this strategy and show empirically that they outperform existing …
Locality Sensitive Hashing (LSH) is an effective method to index a set of points such that we can efficiently find the nearest neighbors of a query point. We extend this method to our novel Set-query LSH (SLSH), such that it can find the nearest neighbors of a set of points, given as a query. Let $ s(x,y) $ be the similarity between two points $ x $ and $ y $. We define a similarity between a set $ Q$ and a point $ x $ by aggregating the similarities $ s(p,x) $ for all $ p\in Q $. For example, we can take $ s(p,x) $ to be the angular similarity between $ p $ and $ x $ (i.e., $1-{\angle (x,p)}/…
Practical Hash Functions for Similarity Estimation and Dimensionality Reduction
University of Copenhagen（哥本哈根大学）；SupWiz（SupWiz公司）
散列是机器学习的几个方面中采用的降维的基本工具。然而，性能分析通常是在使用真正随机的单位代价散列函数的抽象假设下进行的，而不关心使用哪种具体的散列函数。具体的散列函数可以在足够随机输入上工作良好。问题是，当面对更结构化的输入时，它在现实世界中是否值得信任。在本文中，我们专注于散列的两个突出应用，即Li等人的单排列散列（OPH）方案的相似性估计。[NIPS'12]和Weinberger等人的特征散列（FH）。[ICML'09]，这两个都有许多应用，即LSH的近似近邻搜索和SVM的大规模分类。我们认为混合制表哈希Dahlgaard等人。[FOCS'15]，它被证明在许多应用中，包括OPH，表现得像一个真正的随机散列函数。在这里，我们首先展示了使用真正随机散列的FH的改进的浓度范围，然后认为混合制表对稀疏输入的表现类似。然而，我们的主要贡献是在FH，OPH和LSH中使用不同哈希方案的实验比较。我们发现，混合制表散列几乎一样快的multiply-mod-prime方案ax+b mod p. multiply-mod-prime保证工作在足够随机的数据，但我们证明，在上述应用程序中，它可能会导致偏见和现实世界和合成数据的浓度差。我们还比较了流行的MurmurHash 3，它没有经过验证的保证。在我们的实验中，混合制表和MurmurHash 3的表现都类似于真正的随机哈希。然而，混合制表比Mu…
Edit distance similarity search, also called approximate pattern matching, is a fundamental problem with widespread database applications. The goal of the problem is to preprocess $n$ strings of length $d$, to quickly answer queries $q$ of the form: if there is a database string within edit distance $r$ of $q$, return a database string within edit distance $cr$ of $q$. Previous approaches to this problem either rely on very large (superconstant) approximation ratios $c$, or very small search radii $r$. Outside of a narrow parameter range, these solutions are not competitive with trivially sear…
We study the problem of approximate near neighbor (ANN) search and show the following results: - An improved framework for solving the ANN problem using locality-sensitive hashing, reducing the number of evaluations of locality-sensitive hash functions and the word-RAM complexity compared to the standard framework. - A framework for solving the ANN problem with space-time tradeoffs as well as tight upper and lower bounds for the space-time tradeoff of framework solutions to the ANN problem under cosine similarity. - A novel approach to solving the ANN problem on sets along with a matching lowe…
The Jaccard index is an important similarity measure for item sets and Boolean data. On large datasets, an exact similarity computation is often infeasible for all item pairs both due to time and space constraints, giving rise to faster approximate methods. The algorithm of choice used to quickly compute the Jaccard index $\frac{\vert A \cap B \vert}{\vert A\cup B\vert}$ of two item sets $A$ and $B$ is usually a form of min-hashing. Most min-hashing schemes are maintainable in data streams processing only additions, but none are known to work when facing item-wise deletions. In this paper, we …
Similarity-preserving hashing is a core technique for fast similarity searches, and it randomly maps data points in a metric space to strings of discrete symbols (i.e., sketches) in the Hamming space. While traditional hashing techniques produce binary sketches, recent ones produce integer sketches for preserving various similarity measures. However, most similarity search methods are designed for binary sketches and inefficient for integer sketches. Moreover, most methods are either inapplicable or inefficient for dynamic datasets, although modern real-world datasets are updated over time. We…
In recent, deep learning has become the most popular direction in machine learning and artificial intelligence. However, preparation of training data is often a bottleneck in the lifecycle of deploying a deep learning model for production or research. Reusing models for inferencing a dataset can greatly save the human costs required for training data creation. Although there exist a number of model sharing platform such as TensorFlow Hub, PyTorch Hub, DLHub, most of these systems require model uploaders to manually specify the details of each model and model downloaders to screen keyword searc…
