arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.16404cs.PFcs.DB
收藏

在存储上实现与评估 E2LSH

Implementing and Evaluating E2LSH on Storage

Yu Nakanishi, Kazuhiro Hiwada, Yosuke Bando, Tomoya Suzuki, Hirotsugu Kajihara, Shintaro Sano, Tatsuro Endo, Tatsuo Shiozawa

AI总结:

本文提出 E2LSHoS,将 E2LSH 适配至外部存储,利用现代闪存设备评估表明其查询速度超越内存小索引方法,并突破内存索引大小限制实现次线性扩展。

摘要:

局部敏感哈希(LSH)是高维空间中近似最近邻搜索(ANNS)的常用方法之一。针对欧几里得距离的首个 LSH 工作 E2LSH 展示了如何在数据库规模中以次线性查询时间高效解决 ANNS 问题,并具有理论保证的准确性,尽管它需要较大的哈希索引大小。此后,提出了几种具有更小索引大小的 LSH 变体。它们的查询时间是线性或超线性的,但已被证明运行速度更快,因为当索引存储在硬盘驱动器上时它们需要更少的 I/O,并且因为它们也允许在现代 DRAM 容量下进行内存执行。在本文中,我们展示了随着固态硬盘(SSD)等现代闪存设备的出现,E2LSH 正在重新获得查询速度方面的优势。我们在现代单节点计算环境中评估 E2LSH,并分析其计算成本和 I/O 成本,由此推导出其外部内存执行的存储性能要求。我们的分析表明,在单个消费级 SSD 上运行的 E2LSH 比最先进的内存执行小索引方法运行得更快。分析还表明,配备新兴高性能存储设备和接口的 E2LSH 可以接近内存执行 E2LSH 的速度。我们实现了 E2LSH 对外部内存的简单适配,即 E2LSH-on-Storage (E2LSHoS),并使用现代存储设备和接口的不同组合,对多达十亿个对象的实用大型数据集进行了评估。我们证明我们的 E2LSHoS 实现比小索引方法运行得快得多,可以接近内存执行 E2LSH 的速度,并且其查询时间随数据库规模呈次线性扩展,超越了内存执行 E2LSH 的索引大小限制。

英文摘要:

Locality sensitive hashing (LSH) is one of the widely-used approaches to approximate nearest neighbor search (ANNS) in high-dimensional spaces. The first work on LSH for the Euclidean distance, E2LSH, showed how ANNS can be solved efficiently at a sublinear query time in the database size with theoretically-guaranteed accuracy, although it required a large hash index size. Since then, several LSH variants having much smaller index sizes have been proposed. Their query time is linear or superlinear, but they have been shown to run effectively faster because they require fewer I/Os when the index is stored on hard disk drives and because they also permit in-memory execution with modern DRAM capacity. In this paper, we show that E2LSH is regaining the advantage in query speed with the advent of modern flash storage devices such as solid-state drives (SSDs). We evaluate E2LSH on a modern single-node computing environment and analyze its computational cost and I/O cost, from which we derive storage performance requirements for its external memory execution. Our analysis indicates that E2LSH on a single consumer-grade SSD can run faster than the state-of-the-art small-index methods executed in-memory. It also indicates that E2LSH with emerging high-performance storage devices and interfaces can approach in-memory E2LSH speeds. We implement a simple adaptation of E2LSH to external memory, E2LSH-on-Storage (E2LSHoS), and evaluate it for practical large datasets of up to one billion objects using different combinations of modern storage devices and interfaces. We demonstrate that our E2LSHoS implementation runs much faster than small-index methods and can approach in-memory E2LSH speeds, and also that its query time scales sublinearly with the database size beyond the index size limit of in-memory E2LSH.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑