Nearest neighbor (NN) search is inherently computationally expensive in high-dimensional spaces due to the curse of dimensionality. As a well-known solution, locality-sensitive hashing (LSH) is able to answer c-approximate NN (c-ANN) queries in sublinear time with constant probability. Existing LSH methods focus mainly on building hash bucket-based indexing such that the candidate points can be retrieved quickly. However, existing coarse-grained structures fail to offer accurate distance estimation for candidate points, which translates into additional computational overhead when having to examine unnecessary points. This in turn reduces the performance of query processing. In contrast, we propose a fast and accurate in-memory LSH framework, called PM-LSH, that aims to compute c-ANN queries on large-scale, high-dimensional datasets. First, we adopt a simple yet effective PM-tree to index the data points. Second, we develop a tunable confidence interval to achieve accurate distance estimation and guarantee high result quality. Third, we propose an efficient algorithm on top of the PM-tree to improve the performance of computing c-ANN queries. In addition, we extend PM-LSH to support closest pair (CP) search in high-dimensional spaces. We again adopt the PM-tree to organize the points in a lowdimensional space, and we propose a branch and bound algorithm together with a radius pruning technique to improve the performance of computing c-approximate closest pair (c-ACP) queries. Extensive experiments with real-world data offer evidence that PM-LSH is capable of outperforming existing proposals with respect to both efficiency and accuracy for both NN and CP search.
DET-LSH: A Locality-Sensitive Hashing Scheme with Dynamic Encoding Tree for Approximate Nearest Neighbor Search
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);LIPADE, Université Paris Cité(巴黎西岱大学 LIPADE 实验室)
Drawbacks and Proposed Solutions for Real-time Processing on Existing State-of-the-art Locality Sensitive Hashing Techniques
最近邻查询处理是许多图像检索应用的基本操作。通常,图像由特征提取算法生成的高维向量存储和表示。由于基于树的索引结构被证明是无效的高维处理,由于众所周知的“Curse of Precisionality”,近似最近邻技术被用于更快的查询处理。局部敏感哈希(LSH)是一种非常流行和有效的近似最近邻技术,以其次线性查询处理复杂度和理论保证而闻名。如今,随着技术的出现,多个不同的应用领域都需要实时的【形】高维数据存储和处理能力。现有的LSH技术不适合处理实时的【形】数据和查询。在本文中,我们讨论了现有的LSH技术处理实时的【形】高维图像数据的挑战和缺点。此外,通过实验分析,我们提出了改进现有的国家的最先进的LSH技术,有效地处理高维图像数据。
Sampling a Near Neighbor in High Dimensions -- Who is the Fairest of Them All?
IT University of Copenhagen(哥本哈根信息技术大学);University of Illinois at Urbana-Champaign (UIUC)(伊利诺伊大学厄巴纳-香槟分校(UIUC));Toyota Technological Institute at Chicago (TTIC)(芝加哥丰田理工学院(TTIC));University of Copenhagen(哥本哈根大学);University of Padova(帕多瓦大学)