AI 中文总结
本文提出TEngineDB-V，一款腾讯的原生OLAP向量搜索系统，通过全局分块解耦索引等技术解决大k向量搜索问题，实验显示其性能较竞品最高提升145倍，适配百亿级生产部署。
AI 中文摘要
向量搜索系统是现代数据驱动应用的核心基础设施。面向分析的大k向量搜索需返回k=10³至10⁵个结果用于聚合、过滤、连接等分析操作，在腾讯的LLM数据管理、广告分析等新兴 workloads中愈发重要。现有系统仍存在不足：专用向量数据库常为满足尾延迟约束限制k值（如k≤10⁴），且分析支持有限；OLAP系统通常将分块向量索引作为黑盒嵌入，导致严重的读/计算放大，且无法实现原生查询优化。本文提出TEngineDB-V，一款面向大k workloads的原生OLAP向量搜索系统。TEngineDB-V通过以关系表形式物化的全局分块解耦索引，使向量搜索成为腾讯OLAP引擎中的一等分析原语，消除了散点收集执行、降低了放大并支持原生存储优化。它将基于IVFPQ的搜索分解为关系算子，集成OLAP优化，并引入DPPQ——结合方向感知量化与分层残差细化以提升召回率同时保持关系效率。TEngineDB-V还纳入了感知索引的查询重写与感知分布式的成本模型，以实现高效的分布式执行。实验表明，TEngineDB-V相比StarRocks等竞争系统实现了最高145倍的加速，在百亿级规模生产部署中实现了最高52倍的性能提升。
英文摘要
Vector search systems are essential infrastructure for modern data-driven applications. Large-$k$ analytical vector search, which retrieves $k=10^3$--$10^5$ results for analytics (e.g., aggregation, filtering, joins), is increasingly important for emerging workloads, including LLM data management and advertising analysis at Tencent. Existing systems remain inadequate: specialized vector databases often cap $k$ (e.g., $k \leq 10^4$) to satisfy tail-latency constraints and offer limited analytical support, while OLAP systems typically embed per-segment vector indexes as black boxes, causing severe read/compute amplification and preventing native query optimization. This paper presents TEngineDB-V, an OLAP-native vector search system for large-$k$ workloads. TEngineDB-V makes vector search a first-class analytical primitive in Tencent's OLAP engine through a global segment-decoupled index materialized as relational tables, eliminating scatter-gather execution, reducing amplification, and enabling native storage optimizations. It decomposes IVFPQ-based search into relational operators, integrates OLAP optimizations, and introduces DPPQ, which combines direction-aware quantization with hierarchical residual refinement to improve recall while preserving relational efficiency. TEngineDB-V further incorporates index-aware query rewriting and a distributed-aware cost model for efficient distributed execution. Experiments show that TEngineDB-V achieves up to a $145\times$ speedup over competitive systems such as StarRocks, and up to a $52\times$ improvement in 10-billion-scale production deployments.
