AiSAQ:全存储近似最近邻搜索与乘积量化实现无DRAM信息检索
AiSAQ: All-in-Storage ANNS with Product Quantization for DRAM-free Information Retrieval
AI总结:
提出AiSAQ方法,将乘积量化压缩向量卸载至SSD索引,实现十亿级数据集上仅约10MB内存的近似最近邻搜索,且无显著延迟退化,并支持快速索引切换与RAG应用。
摘要:
基于图的近似最近邻搜索(ANNS)算法在大规模向量检索中表现高效。其中,DiskANN通过同时使用DRAM和存储实现了良好的召回率与速度权衡。DiskANN采用乘积量化(PQ)来减少内存使用,但内存使用量仍与数据集规模成正比。本文提出全存储近似最近邻搜索与乘积量化(AiSAQ),将压缩向量卸载到SSD索引中。我们的方法在十亿级数据集上实现查询搜索时约10 MB的内存使用,且无明显延迟退化。AiSAQ还减少了查询搜索准备的索引加载时间,支持在多个十亿级索引间快速切换。该方法可应用于检索增强生成(RAG)的检索器,并可扩展到多服务器系统以应对新兴数据集。我们基于DiskANN的实现已在GitHub上开源。
英文摘要:
Graph-based approximate nearest neighbor search (ANNS) algorithms work effectively against large-scale vector retrieval. Among such methods, DiskANN achieves good recall-speed tradeoffs using both DRAM and storage. DiskANN adopts product quantization (PQ) to reduce memory usage, which is still proportional to the scale of datasets. In this paper, we propose All-in-Storage ANNS with Product Quantization (AiSAQ), which offloads compressed vectors to the SSD index. Our method achieves $\sim$10 MB memory usage in query search with billion-scale datasets without critical latency degradation. AiSAQ also reduces the index load time for query search preparation, which enables fast switch between muitiple billion-scale indices.This method can be applied to retrievers of retrieval-augmented generation (RAG) and be scaled out with multiple-server systems for emerging datasets. Our DiskANN-based implementation is available on GitHub.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。