IRLI：用于学习索引的迭代重新划分
IRLI: Iterative Re-partitioning for Learning to Index
摘要：
神经模型已经改变了基本的信息检索问题，即将查询映射到一个巨大的项目集。然而，对高效和低延迟推理的需求迫使社区重新考虑项目空间中的高效近似近邻搜索。为此，学习索引在最近一段时间获得了很大的兴趣。方法必须在获得高准确性的同时保持负载平衡和分布式环境中的可伸缩性之间进行权衡。我们提出了一种新的方法，称为IRLI（发音为“早”），迭代分区的项目通过学习直接从查询项目的相关性数据的相关桶。此外，IRLI采用基于$k$选择功率的高级负载均衡策略。我们从数学上表明，IRLI检索正确的项目，在非常自然的假设下，具有很高的概率，并提供优越的负载均衡。IRLI在多标签分类上超过了最佳基线的精度，同时在推理上比$5x$更快。对于近邻搜索任务，相同的方法优于最先进的学习散列方法NeuralLSH，因为对于相同的召回仅需要约{1/6}^th的候选。IRLI是数据和模型并行的，使其成为分布式GPU实现的理想选择。我们通过索引1亿个密集向量并在召回率上超过流行的FAISS库10%来证明这一优势。
英文摘要：
Neural models have transformed the fundamental information retrieval problem of mapping a query to a giant set of items. However, the need for efficient and low latency inference forces the community to reconsider efficient approximate near-neighbor search in the item space. To this end, learning to index is gaining much interest in recent times. Methods have to trade between obtaining high accuracy while maintaining load balance and scalability in distributed settings. We propose a novel approach called IRLI (pronounced `early'), which iteratively partitions the items by learning the relevant buckets directly from the query-item relevance data. Furthermore, IRLI employs a superior power-of-$k$-choices based load balancing strategy. We mathematically show that IRLI retrieves the correct item with high probability under very natural assumptions and provides superior load balancing. IRLI surpasses the best baseline's precision on multi-label classification while being $5x$ faster on inference. For near-neighbor search tasks, the same method outperforms the state-of-the-art Learned Hashing approach NeuralLSH by requiring only ~ {1/6}^th of the candidates for the same recall. IRLI is both data and model parallel, making it ideal for distributed GPU implementation. We demonstrate this advantage by indexing 100 million dense vectors and surpassing the popular FAISS library by >10% on recall.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
