用于高维大规模多媒体数据集相似性搜索的可扩展局部敏感哈希算法
Scalable Locality-Sensitive Hashing for Similarity Search in High-Dimensional, Large-Scale Multimedia Datasets
摘要：
相似性搜索对于许多数据库应用至关重要，包括日益流行的基于内容的多媒体检索（CBMR）在线服务。这些服务，包括图像搜索引擎，必须处理大量的数据，同时保持较低的响应时间。因此，可扩展性是必不可少的相似性搜索在Web规模的应用程序，但大多数现有的方法是顺序和目标共享内存的机器。在这里，我们解决了这些问题，一个分布式的，高效的，可扩展的索引基于局部敏感哈希（LSH）。LSH是最有效和最流行的相似性搜索技术之一，但其不良的引用局部性使其实现具有挑战性的问题。我们的解决方案是基于一个广泛的异步并行化，具有许多优化，包括分层并行化解耦索引和数据存储，本地感知的数据分区策略，以减少消息传递，和多探测，以限制内存使用。建议的并行化达到了90%的效率，在分布式系统约800个CPU核心。特别是，原始的本地感知数据分区将交换的消息数量减少了30%。我们的并行LSH使用最大的公共数据集进行相似性搜索（据我们所知），从Web图像中提取$10^9$ 128-d SIFT描述符。这比以前的LSH并行化可以处理的数据集大两个数量级。
英文摘要：
Similarity search is critical for many database applications, including the increasingly popular online services for Content-Based Multimedia Retrieval (CBMR). These services, which include image search engines, must handle an overwhelming volume of data, while keeping low response times. Thus, scalability is imperative for similarity search in Web-scale applications, but most existing methods are sequential and target shared-memory machines. Here we address these issues with a distributed, efficient, and scalable index based on Locality-Sensitive Hashing (LSH). LSH is one of the most efficient and popular techniques for similarity search, but its poor referential locality properties has made its implementation a challenging problem. Our solution is based on a widely asynchronous dataflow parallelization with a number of optimizations that include a hierarchical parallelization to decouple indexing and data storage, locality-aware data partition strategies to reduce message passing, and multi-probing to limit memory usage. The proposed parallelization attained an efficiency of 90% in a distributed system with about 800 CPU cores. In particular, the original locality-aware data partition reduced the number of messages exchanged in 30%. Our parallel LSH was evaluated using the largest public dataset for similarity search (to the best of our knowledge) with $10^9$ 128-d SIFT descriptors extracted from Web images. This is two orders of magnitude larger than datasets that previous LSH parallelizations could handle.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
