ProMIPS:高效的高维c-近似最大内积搜索
ProMIPS: Efficient High-Dimensional c-Approximate Maximum Inner Product Search with a Lightweight Index
摘要:
由于在推荐系统、多类标签预测和深度学习等领域的广泛应用,近年来最大内积(MIP)搜索问题受到了广泛关注。面对包含高维特征向量的大规模数据集,现有的基于LSH的方法通常需要使用大量的哈希表或长哈希码来保证搜索质量,这会占用大量的索引空间并导致过多的磁盘页面访问。在本文中,我们放松了效率的准确性的保证,并提出了一个有效的方法c-近似最大内积(c-AMIP)搜索与轻量级iDistance索引。我们通过2-稳定随机投影将高维点投影到低维点上,并推导出保证概率的搜索条件,从而保证c-AMIP结果在任意概率下的精度.为了进一步提高效率,我们提出了快速探测,快速确定搜索范围满足推导条件提前,避免了低效的增量搜索过程。在四个真实数据集上的实验结果表明,该方法需要更少的预处理成本,包括索引大小和预处理时间。此外,与最先进的基准方法相比,它在整体比率和召回率方面的搜索质量以及页面访问和运行时间方面的效率方面提供了优越的结果。
英文摘要:
Due to the wide applications in recommendation systems, multi-class label prediction and deep learning, the Maximum Inner Product (MIP) search problem has received extensive attention in recent years. Faced with large-scale datasets containing high-dimensional feature vectors, the state-of-the-art LSH-based methods usually require a large number of hash tables or long hash codes to ensure the searching quality, which takes up lots of index space and causes excessive disk page accesses. In this paper, we relax the guarantee of accuracy for efficiency and propose an efficient method for c-Approximate Maximum Inner Product (c-AMIP) search with a lightweight iDistance index. We project high-dimensional points to low-dimensional ones via 2-stable random projections and derive probability-guaranteed searching conditions, by which the c-AMIP results can be guaranteed in accuracy with arbitrary probabilities. To further improve the efficiency, we propose Quick-Probe for quickly determining the searching bound satisfying the derived condition in advance, avoiding the inefficient incremental searching process. Extensive experimental evaluations on four real datasets demonstrate that our method requires less pre-processing cost including index size and pre-processing time. In addition, compared to the state-of-the-art benchmark methods, it provides superior results on searching quality in terms of overall ratio and recall, and efficiency in terms of page access and running time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。