PQTable:基于哈希表的乘积量化码非穷举快速搜索
PQTable: Non-exhaustive Fast Search for Product-quantized Codes using Hash Tables
摘要:
在本文中,我们提出了一个积量化表(PQTable),一个快速搜索方法的积量化码通过哈希表。每个数据库向量的标识符通过使用其PQ码作为关键字而与哈希表的槽相关联。对于查询,输入向量被PQ编码和散列,然后检索与该代码相关联的项。建议的PQ表产生与线性PQ扫描相同的结果,并且快10^2到10^5倍。虽然通过以前的基于反向索引的方法可以实现最先进的性能,但这些方法需要手动设计的参数设置和大量的培训;我们的PQTable没有这些限制,因此为现实世界的问题提供了实用有效的解决方案。具体来说,当向量被高度压缩时,我们的PQTable在单个CPU上实现了迄今为止最快的搜索性能之一,并且内存使用效率非常高(在10^9个数据点上每次查询0.059 ms,仅消耗5.5 GB内存)。最后,我们表明,我们提出的PQTable可以自然地处理优化的产品量化(OPQTable)的代码。
英文摘要:
In this paper, we propose a product quantization table (PQTable); a fast search method for product-quantized codes via hash-tables. An identifier of each database vector is associated with the slot of a hash table by using its PQ-code as a key. For querying, an input vector is PQ-encoded and hashed, and the items associated with that code are then retrieved. The proposed PQTable produces the same results as a linear PQ scan, and is 10^2 to 10^5 times faster. Although state-of-the-art performance can be achieved by previous inverted-indexing-based approaches, such methods require manually-designed parameter setting and significant training; our PQTable is free of these limitations, and therefore offers a practical and effective solution for real-world problems. Specifically, when the vectors are highly compressed, our PQTable achieves one of the fastest search performances on a single CPU to date with significantly efficient memory usage (0.059 ms per query over 10^9 data points with just 5.5 GB memory consumption). Finally, we show that our proposed PQTable can naturally handle the codes of an optimized product quantization (OPQTable).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。