FreshDiskANN：一种快速准确的基于图的ANN流媒体相似性搜索索引
FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search
摘要：
近似最近邻搜索（ANNS）是信息检索中的基本构建块，其中基于图的索引是当前最先进的并且在工业中广泛使用。基于图的索引的最新进展使得在具有SSD的单个商品机器上索引和搜索具有高召回率和毫秒级延迟的十亿点数据集成为可能。然而，现有的人工神经网络图算法只支持静态索引，不能反映许多关键的现实世界场景所需的语料库的实时变化（例如，文档中的句子索引，电子邮件或新闻索引）。为了克服这个缺点，当前用于将更新体现到这样的索引中的行业实践是周期性地重新构建这些索引，这可能是非常昂贵的。在本文中，我们提出了第一个基于图的ANNS索引，该索引将语料库更新实时反映到索引中，而不会影响搜索性能。使用此索引的更新规则，我们设计了FreshmartANN，一个可以在具有SSD和有限内存的工作站上索引超过10亿个点的系统，并支持每秒数千次并发实时插入，删除和搜索，同时保留$>95\%$ 5-recall@5。与现有方法相比，这意味着保持索引新鲜度的成本降低了5- 10倍。
英文摘要：
Approximate nearest neighbor search (ANNS) is a fundamental building block in information retrieval with graph-based indices being the current state-of-the-art and widely used in the industry. Recent advances in graph-based indices have made it possible to index and search billion-point datasets with high recall and millisecond-level latency on a single commodity machine with an SSD. However, existing graph algorithms for ANNS support only static indices that cannot reflect real-time changes to the corpus required by many key real-world scenarios (e.g. index of sentences in documents, email, or a news index). To overcome this drawback, the current industry practice for manifesting updates into such indices is to periodically re-build these indices, which can be prohibitively expensive. In this paper, we present the first graph-based ANNS index that reflects corpus updates into the index in real-time without compromising on search performance. Using update rules for this index, we design FreshDiskANN, a system that can index over a billion points on a workstation with an SSD and limited memory, and support thousands of concurrent real-time inserts, deletes and searches per second each, while retaining $>95\%$ 5-recall@5. This represents a 5-10x reduction in the cost of maintaining freshness in indices when compared to existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
