arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.04435cs.CLcs.DS
收藏

用于最近邻查找的快速精确检索(FERN)

Fast Exact Retrieval for Nearest-neighbor Lookup (FERN)

Princeton University(普林斯顿大学)

Richard Zhu

AI总结:

针对高维精确最近邻搜索复杂度高的问题,本文受 kd-trees 启发提出 FERN 算法,在千万个 d=128 随机向量上实现 O(d log N) 查找与 100% 召回率。

摘要:

精确最近邻搜索是一个计算密集型过程,即使是其较简单的同类问题——向量检索——也可能具有很高的计算复杂度。当所检索向量的维度 d 相对于数据库中的向量数量 N 较高时,这一问题会进一步恶化。精确最近邻检索被普遍认为是一个 O(Nd) 问题,不存在亚线性解决方案。因此,注意力转向了近似最近邻(ANN)检索技术,其中许多技术具有亚线性甚至对数时间复杂度。然而,如果我们从二分搜索问题(例如 d=1 的向量检索)中获得的直觉仍然成立,那么应当存在一种方法,可在不通过暴力搜索求解的情况下检索向量的有组织表示。对于低维情形(例如 d=2 或 d=3),kd-trees 提供了一种 O(d log N) 的检索算法。遗憾的是,在高维情形下(例如 k=128),该算法在实践中会迅速退化为 O(dN) 的解决方案。我们受 kd-trees 启发,提出了一种用于对数级快速精确最近邻查找(Fast Exact Retrieval for Nearest-neighbor lookup,FERN)的新颖算法。该算法在 1000 万个 d=128 的均匀随机生成向量上实现了 O(d log N) 查找,并取得 100% 召回率。代码可在 https://github.com/RichardZhu123/ferns 获取。

英文摘要:

Exact nearest neighbor search is a computationally intensive process, and even its simpler sibling -- vector retrieval -- can be computationally complex. This is exacerbated when retrieving vectors which have high-dimension $d$ relative to the number of vectors, $N$, in the database. Exact nearest neighbor retrieval has been generally acknowledged to be a $O(Nd)$ problem with no sub-linear solutions. Attention has instead shifted towards Approximate Nearest-Neighbor (ANN) retrieval techniques, many of which have sub-linear or even logarithmic time complexities. However, if our intuition from binary search problems (e.g. $d=1$ vector retrieval) carries, there ought to be a way to retrieve an organized representation of vectors without brute-forcing our way to a solution. For low dimension (e.g. $d=2$ or $d=3$ cases), \texttt{kd-trees} provide a $O(d\log N)$ algorithm for retrieval. Unfortunately the algorithm deteriorates rapidly to a $O(dN)$ solution at high dimensions (e.g. $k=128$), in practice. We propose a novel algorithm for logarithmic Fast Exact Retrieval for Nearest-neighbor lookup (FERN), inspired by \texttt{kd-trees}. The algorithm achieves $O(d\log N)$ look-up with 100\% recall on 10 million $d=128$ uniformly randomly generated vectors.\footnote{Code available at https://github.com/RichardZhu123/ferns}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑