arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.01231cs.DB
收藏

重新审视基于邻近图的近似最近邻搜索的索引构建

Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search

Shuo Yang, Jiadong Xie, Yingfan Liu, Jeffrey Xu Yu, Xiyue Gao, Qianru Wang, Yanguo Peng, Jiangtao Cui

AI总结:

本文重新审视 RNG 与 NSWG 的索引构建,提出含新型边选择剪枝策略的框架,在不损失 $k$-ANN 搜索性能的同时最高实现 5.6 倍构建加速。

摘要:

邻近图(PG)作为高维数据上 $k$-近似最近邻($k$-ANN)搜索的最先进解决方案而日益普及,该搜索是检索增强生成等众多领域中的一项基础功能。尽管基于 PG 的方法具有最佳的 $k$-ANN 搜索性能,但其索引构建成本相对于点数呈超线性增长。这种超线性成本在大数据时代严重限制了其可扩展性。因此,本文的目标是在不损害其 $k$-ANN 搜索性能的情况下,加速基于 PG 的方法的构建。为实现这一目标,本文重新审视了两类主流 PG:相对邻域图(RNG)和可导航小世界图(NSWG)。通过重新审视它们的构建过程,我们发现了构建效率方面的问题。为解决这些问题,我们提出了一个新的构建框架,该框架采用一种用于边选择的新型剪枝策略,可在保持 RNG 的 $k$-ANN 搜索性能的同时加速 RNG 构建。随后,我们将该框架集成到 NSWG 构建中,以同时提升 NSWG 的构建效率和 $k$-ANN 搜索性能。我们开展了大量实验,以验证该构建框架对 RNG 和 NSWG 均有效,并且它显著降低了 PG 构建成本,实现了最高 5.6 倍的加速,同时没有损害 $k$-ANN 搜索性能。

英文摘要:

Proximity graphs (PG) have gained increasing popularity as the state-of-the-art solutions to $k$-approximate nearest neighbor ($k$-ANN) search on high-dimensional data, which serves as a fundamental function in various fields, e.g., retrieval-augmented generation. Although PG-based approaches have the best $k$-ANN search performance, their index construction cost is superlinear to the number of points. Such superlinear cost substantially limits their scalability in the era of big data. Hence, the goal of this paper is to accelerate the construction of PG-based methods without compromising their $k$-ANN search performance. To achieve this goal, two mainstream categories of PG are revisited: relative neighborhood graph (RNG) and navigable small world graph (NSWG). By revisiting their construction process, we find the issues of construction efficiency. To address these issues, we propose a new construction framework with a novel pruning strategy for edge selection, which accelerates RNG construction while keeping its $k$-ANN search performance. Then, we integrate this framework into NSWG construction to enhance both the construction efficiency and $k$-ANN search performance of NSWG. Extensive experiments are conducted to validate our construction framework for both RNG and NSWG, and that it significantly reduces the PG construction cost, achieving up to 5.6x speedup, while not compromising the $k$-ANN search performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑