arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.09426cs.IR
收藏

LANNS:一个网络规模的近似最近邻查询系统

LANNS: A Web-Scale Approximate Nearest Neighbor Lookup System

Ishita Doshi, Dhritiman Das, Ashish Bhutani, Rajeev Kumar, Rushi Bhatt, Niranjan Balasubramanian

摘要:

最近邻搜索在信息检索、计算机视觉、机器学习、数据库等领域有着广泛的应用。现有的最先进的最近邻搜索算法,分层可导航小世界网络(HNSW),是无法扩展到大数据集的1亿条记录的高维数据。在本文中,我们提出了LANS,这是一个用于近似最近邻搜索的端到端平台,可用于网络规模的数据集。用于大规模近似最近邻搜索的库(LANS)被部署在多个生产系统中,用于识别topK($100 \leq topK \leq 200$)近似最近邻,每个查询的延迟为几毫秒,单个节点上的高吞吐量为2.5k字节每秒(QPS),在大型($\sim$ 180 M数据点)高维(50-2048维)数据集上。

英文摘要:

Nearest neighbor search (NNS) has a wide range of applications in information retrieval, computer vision, machine learning, databases, and other areas. Existing state-of-the-art algorithm for nearest neighbor search, Hierarchical Navigable Small World Networks(HNSW), is unable to scale to large datasets of 100M records in high dimensions. In this paper, we propose LANNS, an end-to-end platform for Approximate Nearest Neighbor Search, which scales for web-scale datasets. Library for Large Scale Approximate Nearest Neighbor Search (LANNS) is deployed in multiple production systems for identifying topK ($100 \leq topK \leq 200$) approximate nearest neighbors with a latency of a few milliseconds per query, high throughput of 2.5k Queries Per Second (QPS) on a single node, on large ($\sim$180M data points) high dimensional (50-2048 dimensional) datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑