发表机构
Nanyang Technological University; ETH Zurich; Meta（南洋理工大学; 苏黎世联邦理工学院; Meta）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对SSD驻留向量搜索，提出RaBitQ-SSD，采用分裂码与流水线I/O，减少SSD读取并隐藏延迟，在90%召回率下吞吐量提升1.74倍，索引更小更快。
AI 中文摘要
当向量集合超过DRAM容量时，SSD驻留的近似最近邻搜索至关重要。挑战在于减少SSD读取，并通过并发读取和与计算重叠来隐藏I/O延迟。然而，对于基于图的搜索，渐进式候选发现限制了提前进行I/O规划，而IVF搜索则完整读取倒排列表，导致不必要的SSD读取。在本文中，我们提出了RaBitQ-SSD，它是IVF-RaBitQ的扩展，用于SSD驻留向量搜索。具体来说，我们提出了Split-RaBitQ，它在DRAM中保留每个二进制码的可配置前缀，使得内存中的码存储低于每维度一比特。利用这种部分表示，它提供了一个具有概率误差界限的无偏距离估计器。一旦内存中的粗量化器识别出候选列表，该界限支持对其中单个候选进行剪枝，从而实现更细粒度的SSD访问。我们还设计了一个异步搜索流水线，将候选剪枝与SSD读取调度协调起来，以减少不必要的读取，同时将I/O与计算重叠。在从500万到10亿向量的数据集上，RaBitQ-SSD在90%召回率下提供了高达图基基线1.74倍的吞吐量，同时将SSD页读取减少了高达3.8倍。其SSD上的索引比DiskANN小高达7.0倍，构建速度快高达10.0倍。我们还在单台机器上使用一个SSD构建并搜索了100亿向量的索引，在90%召回率下实现了每秒超过3000次查询。
英文摘要
SSD-resident approximate nearest-neighbor search is essential when vector collections exceed DRAM capacity. The challenge is to reduce SSD reads and hide I/O latency through concurrent reads and overlap with computation. However, for graph-based search, progressive candidate discovery limits advance I/O planning while IVF search reads inverted lists in full, incurring unnecessary SSD reads. In this paper, we present RaBitQ-SSD, an extension of IVF-RaBitQ for SSD-resident vector search. Specifically, we propose Split-RaBitQ, which retains a configurable prefix of each binary code in DRAM, allowing in-memory code storage below one bit per dimension. Using this partial representation, it provides an unbiased distance estimator with a probabilistic error bound. Once the in-memory coarse quantizer identifies candidate lists, this bound supports pruning individual candidates within them, enabling finer-grained SSD access. We also design an asynchronous search pipeline that coordinates candidate pruning with SSD read scheduling to reduce unnecessary reads while overlapping I/O with computation. On datasets ranging from $5$ million to $1$ billion vectors, RaBitQ-SSD delivers up to $1.74\times$ the throughput of graph-based baselines at $90\%$ recall, while reducing SSD page reads by up to $3.8\times$. Its on-SSD indexes are up to $7.0\times$ smaller and $10.0\times$ faster to build than DiskANN. We also build and search an index of $10$ billion vectors on a single machine with one SSD, achieving over $3{,}000$ queries per second at $90\%$ recall.
