向量数据库中的近似多样性 $k$-近邻搜索
Approximate Diverse $k$-nearest Neighbor Search in Vector Database
AI总结:
本文提出一种将结果多样性无缝集成到SOTA A$k$-NNS方法的渐进式搜索框架,通过迭代搜索、多样化和验证阶段,在无额外索引开销下高效逼近最优多样性结果集,在三个百万级数据集上展现了低延迟与高召回的优越性。
摘要:
近似 $k$-近邻搜索(A$k$-NNS)是向量数据库中的核心操作,支撑着检索增强生成(RAG)和图像检索等应用。在这些场景中,用户通常偏好具有多样性的结果集,以最小化冗余并提升信息价值。然而,现有的基于贪心的多样性方法通常产生次优结果,无法在特定多样性水平下充分逼近最优相似度得分。此外,还需要能够适应不同用户定义结果大小和多样性需求的灵活算法。为了应对这些挑战,我们提出了一种新颖的方法,将结果多样性无缝集成到最先进(SOTA)的 A$k$-NNS 方法中。我们的方法引入了渐进式搜索框架,包含迭代搜索、多样化和验证阶段。经过精心设计的多样化和验证步骤,使我们的方法能够根据用户指定的多样性水平,在无需额外索引开销的情况下高效逼近最优多样性结果集。我们在三个百万规模的基准数据集 LAION-art、Deep1M 和 Txt2img 上评估了我们的方法,使用延迟、相似度和召回率作为性能指标,涵盖了不同的 $k$ 值和多样性阈值。实验结果表明,我们的方法始终能以最小的延迟开销检索到接近最优的多样性结果,特别是在中高多样性设置下。
英文摘要:
Approximate $k$-nearest neighbor search (A$k$-NNS) is a core operation in vector databases, underpinning applications such as retrieval-augmented generation (RAG) and image retrieval. In these scenarios, users often prefer diverse result sets to minimize redundancy and enhance information value. However, existing greedy-based diverse methods frequently yield sub-optimal results, failing to adequately approximate the optimal similarity score under certain diversification level. Furthermore, there is a need for flexible algorithms that can adapt to varying user-defined result sizes and diversity requirements. To address these challenges, we propose a novel approach that seamlessly integrates result diversification into state-of-the-art (SOTA) A$k$-NNS methods. Our approach introduces a progressive search framework, consisting of iterative searching, diversification, and verification phases. Carefully designed diversification and verification steps enable our approach to efficiently approximate the optimal diverse result set according to user-specified diversification levels without additional indexing overhead. We evaluate our method on three million-scale benchmark datasets, LAION-art, Deep1M, and Txt2img, using latency, similarity, and recall as performance metrics across a range of $k$ values and diversification thresholds. Experimental results demonstrate that our approach consistently retrieves near-optimal diverse results with minimal latency overhead, particularly under medium and high diversity settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。