FluxSieve: 统一流式与分析数据平面以实现可扩展的云可观测性
FluxSieve: Unifying Streaming and Analytical Data Planes for Scalable Cloud Observability
AI总结：
FluxSieve通过统一流式与分析数据平面，实现高并发下查询性能的显著提升，同时保持低存储和计算开销。
摘要：
尽管在查询优化、索引技术和数据存储方面取得了许多进展，现代数据平台仍然在高并发和计算密集型查询下难以提供稳健的查询性能。这一挑战在处理高体积、高速度数据记录的大型可观测性平台中尤为突出。例如，在查询时间进行的重复、昂贵的过滤查询会对分析数据平面造成显著的计算和存储开销。在本文中，我们提出FluxSieve，一种统一的架构，通过在数据摄入路径中嵌入一个轻量级的流内预计算和过滤层，将传统的拉式查询处理与推式流处理相结合。这避免了在专用流处理框架中运行查询的复杂性和操作负担。具体而言，这项工作（i）介绍了一种基础架构，通过流内过滤和记录增强统一流式和分析数据平面；（ii）设计了一种可扩展的多模式匹配机制，支持同时评估和实时更新过滤规则，具有极低的每条记录开销；（iii）展示了如何将这种摄入时间处理与两个开源分析系统整合——Apache Pinot作为实时在线分析处理（RTOLAP）引擎和DuckDB作为嵌入式分析数据库；（iv）对我们的方法进行了全面的实验评估。我们的评估在不同系统、查询类型和性能指标上显示，查询性能可提升多个数量级，代价是极小的额外存储和非常低的计算开销。
英文摘要：
Despite many advances in query optimization, indexing techniques, and data storage, modern data platforms still face difficulties in delivering robust query performance under high concurrency and computationally intensive queries. This challenge is particularly pronounced in large-scale observability platforms handling high-volume, high-velocity data records. For instance, recurrent, expensive filtering queries at query time impose substantial computational and storage overheads in the analytical data plane. In this paper, we propose FluxSieve, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path. This avoids the complexity and operational burden of running queries in dedicated stream processing frameworks. Concretely, this work (i) introduces a foundational architecture that unifies streaming and analytical data planes via in-stream filtering and records enrichment, (ii) designs a scalable multi-pattern matching mechanism that supports concurrent evaluation and on-the-fly updates of filtering rules with minimal per-record overhead, (iii) demonstrates how to integrate this ingestion-time processing with two open-source analytical systems -- Apache Pinot as a Real-Time Online Analytical Processing (RTOLAP) engine and DuckDB as an embedded analytical database, and (iv) performs comprehensive experimental evaluation of our approach. Our evaluation across different systems, query types, and performance metrics shows up to orders-of-magnitude improvements in query performance at the cost of negligible additional storage and very low computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
