arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1603.09320cs.DScs.CVcs.IRcs.SI
收藏

基于分层可导航小世界图的高效鲁棒近似最近邻搜索

Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs

Institute of Applied Physics of the Russian Academy of Sciences(俄罗斯科学院应用物理研究所)

Yu. A. Malkov, D. A. Yashunin

摘要:

本文提出了一种基于可控层次可导航小世界图(Hierarchical NSW,HNSW)的近似K近邻搜索方法。所提出的解决方案是完全基于图的,而不需要任何额外的搜索结构,这通常是在最接近图技术的粗搜索阶段使用。分层NSW递增地构建由用于所存储元素的嵌套子集的邻近图(层)的分层集合组成的多层结构。其中存在元素的最大层以指数衰减概率分布随机选择。这允许产生类似于先前研究的可导航小世界(NSW)结构的图形,同时还具有由其特征距离尺度分开的链接。与NSW相比,从上层开始搜索并利用尺度分离提高了性能,并允许对数复杂度缩放。额外的就业的启发式选择邻近图邻居显着提高性能在高召回率和高度聚类的数据的情况下。性能评估表明,所提出的通用度量空间搜索索引能够大大优于以前的开源最先进的矢量方法。该算法与跳表结构的相似性允许简单的平衡分布式实现。

英文摘要:

We present a new approach for the approximate K-nearest neighbor search based on navigable small world graphs with controllable hierarchy (Hierarchical NSW, HNSW). The proposed solution is fully graph-based, without any need for additional search structures, which are typically used at the coarse search stage of the most proximity graph techniques. Hierarchical NSW incrementally builds a multi-layer structure consisting from hierarchical set of proximity graphs (layers) for nested subsets of the stored elements. The maximum layer in which an element is present is selected randomly with an exponentially decaying probability distribution. This allows producing graphs similar to the previously studied Navigable Small World (NSW) structures while additionally having the links separated by their characteristic distance scales. Starting search from the upper layer together with utilizing the scale separation boosts the performance compared to NSW and allows a logarithmic complexity scaling. Additional employment of a heuristic for selecting proximity graph neighbors significantly increases performance at high recall and in case of highly clustered data. Performance evaluation has demonstrated that the proposed general metric space search index is able to strongly outperform previous opensource state-of-the-art vector-only approaches. Similarity of the algorithm to the skip list structure allows straightforward balanced distributed implementation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑