通过拍卖进行向量搜索的聚类
Cluster with Auctions for Vector Search
浏览论文内容
中文总结 AI 辅助
研究针对大规模近似最近邻搜索中查询探测函数和数据库分区未分开处理的问题，提出CwA方法，联合学习平衡的数据库分区与神经探测函数，通过交替两步优化搜索性能，扩展后在不同分布下表现出色，提升了吞吐量。
中文摘要 AI 辅助
大规模近似最近邻搜索通常依赖分区进行索引，数据库向量被划分为簇，每个查询通过探测函数选择要扫描的簇。查询探测函数和数据库分区很少被视为独立实体，多数技术为查询和数据库向量使用相同分配函数，这在数据库和查询分布不同时欠佳。本文介绍了CwA（通过拍卖进行聚类），通过联合学习平衡的数据库分区和神经探测函数解决此限制。CwA直接针对查询分布优化搜索性能，通过交替两步最小化目标：一是对探测函数神经网络进行梯度下降，二是对数据库向量的簇分配进行大规模组合优化，用可并行的拍卖算法解决后者以平衡分区。为进一步扩展CwA，将方法扩展到簇的笛卡尔积以增加分区粒度。当数据库和查询分布不同时，CwA在相同召回率下吞吐量比现有技术高4.7倍。在分布内设置中，即使是用CwA训练的简单线性探测函数也优于竞争的深度神经方法。
英文摘要
Large-scale approximate nearest neighbor search commonly relies on partitions for indexing: database vectors are partitioned into clusters, and for each query a probing function selects the clusters to be scanned. The query probing function and the database partition are rarely treated as separate entities: most techniques assign queries with the same assignment function as the database vectors, which is suboptimal especially when database and query distributions differ. This paper introduces CwA (Cluster with Auctions), which addresses this limitation by jointly learning a balanced database partition and a neural probing function. CwA optimizes search performance directly for the query distribution. It minimizes its objective by alternating two steps: (i) gradient descent on the neural network of the probing function, and (ii) a large-scale combinatorial optimization of the cluster assignment for the database vectors. We solve the latter with a parallelizable auction algorithm that balances the partition by design. To further scale CwA, we extend the method to a Cartesian product of clusters that increases the partition's granularity. When database and query distributions differ, CwA achieves up to 4.7$\times$ throughput over the state-of-the-art at equal recall. In the in-distribution (ID) setting, even a simple linear probing function trained with CwA outperforms competing deep neural methods.
