UBASE：字节跳动万亿级向量数据管理的AI搜索引擎

中文摘要 AI 辅助

自2016年起，UBASE已成为字节跳动搜索基础设施的核心，扩展至超过7000个集群和300PB的索引数据。受AI工作负载需求驱动，UBASE已从文本搜索引擎演变为统一的AI搜索系统，支持向量检索、词汇匹配和谓词过滤。其最大部署规模可索引近万亿个高维向量。该规模暴露了AI时代检索的两个核心瓶颈：持续数据摄入下内存密集型的图索引构建，以及将向量索引完全驻留内存的高昂成本。UBASE通过两项技术解决这些瓶颈：第一，引入基于SymRaBitQ的量化感知向量内核，SymRaBitQ是一种具有严格理论保证的新型对称量化方案，可让索引构建直接在量化空间中准确高效地运行，无需保留全精度向量的副本；第二，提供混合存储引擎，支持内存驻留、混合及SSD驻留部署，具备细粒度记录级缓存，可在操作控制下以内存换取延迟。在大规模基准测试中，与现有系统相比，UBASE的吞吐量提升最高达3倍，索引内存减少80%，运营成本降低86%，同时支持万亿级向量规模、写密集型或延迟敏感的生产工作负载。

英文摘要

Since 2016, ByteX has been the foundation of ByteDance's search infrastructure, scaling to more than 7,000 clusters and 300 PB of indexed data. Driven by the demands of AI workloads, ByteX has evolved from a text search engine into a unified AI search system supporting vector retrieval, lexical matching, and predicate filtering. Its largest deployment indexes nearly one trillion high-dimensional vectors. This scale exposes two central bottlenecks in AI-era retrieval: memory-intensive graph-index construction under sustained ingestion, and the prohibitive cost of keeping vector indexes entirely in memory. ByteX addresses these bottlenecks with two techniques. First, it introduces a quantization-aware vector kernel based on SymRaBitQ, a new symmetric quantization scheme with tight theoretical guarantees that allows index construction to run directly in the quantized space accurately and efficiently without retaining a copy of full-precision vectors. Second, it provides a hybrid storage engine that supports memory-resident, hybrid, and SSD-resident deployments, with fine-grained record-level caching to trade memory for latency under operational control. On large-scale benchmarks, ByteX improves throughput by up to 3x, reduces indexing memory by 80%, and lowers operating cost by 86% compared with prior systems, while supporting trillion-vector scale, write-heavy or latency-sensitive workloads in production.

发表机构

• ByteDance（字节跳动）

机构由 AI 辅助整理，请以论文原文为准。