AI 中文总结
研究基于图的ANN搜索中查询硬度预测问题,提出SHEAF度量标准,通过答案集通量表示查询硬度,设计自剖析估计器和评估协议,在多个数据集和索引上,相比基线度量能更好地预测查询束宽,提升预测效果。
AI 中文摘要
基于图的近似近邻(ANN)搜索通常由一个束宽参数控制,该参数在召回率和吞吐量之间进行权衡且在整个工作负载中固定。然而,查询难度可能不同,如在SIFT1M数据集上,达到95%召回率所需的束宽变化超过32倍。当前衡量查询难度的代理指标本地内在维度(LID)存在局限性。本文提出了一种新的度量标准SHEAF,它通过查询自身前k个答案集在两个浅探测宽度之间的变化来表示查询的硬度。设计了自剖析估计器将通量转化为可部署的每个查询的束预测器,还开发了固定探测评估协议。在四个不同数据集上的流行ANN索引(如CAGRA和HNSW)上,SHEAF在GPU和CPU上预测每个查询的束比五个基线度量更好,在留出相关性方面提高了1.55倍,且仅使用两次浅探测搜索且无需查询时的真实情况。
英文摘要
Graph-based approximate nearest neighbor (ANN) search is usually governed by a beam-width parameter that trades recall for throughput and is fixed for the whole workload. Yet, queries may not be equally hard: for example, on the widely used data set SIFT1M, the beam that a query needs to reach 95\% recall varies by more than $32\times$. Therefore, serving each query at its own width would help if the system could tell, cheaply and in advance, how hard it is. The prevailing proxy for this difficulty is called local intrinsic dimensionality (LID); however, LID is static and geometric, which makes it only weakly predict the minimum beam. This paper presents a new measure, namely Self-profiled Hardness Estimation from Answer-set Flux (SHEAF), which represents a query's hardness as how much its own top-$k$ answer set changes between two shallow probe widths. We design a self-profiling estimator that turns this flux into a deployable per-query beam predictor; furthermore, we develop a fixed-probe evaluation protocol that scores each measure over all queries with an observed minimum sufficient beam. On popular ANN indexes such as CAGRA and HNSW across four diverse data sets, SHEAF predicts the per-query beam better than five baseline measures on both GPU and CPU by up to $1.55\times$ in held-out correlation, using only two shallow probe searches and no query-time ground truth.