arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05462cs.DB
收藏

MatrixGate:一种面向时序数据库的高性能数据摄取工具

MatrixGate: A High-performance Data Ingestion Tool for Time-series Databases

Shuhui Wang, Zihan Sun, Chaochen Hu, Chao Li, Yong Zhang, Yandong Yao, Hao Wang, Chunxiao Xing

AI总结:

本文提出 MatrixGate 时序数据摄取工具,通过 slots 调优、多协程和无锁队列提升并行度与效率,加载速度比现有方法快 3-100 倍,查询延迟降低约 80%,并具备 86% 的分布式可扩展性。

摘要:

近年来,众多领域产生了海量时序数据。这种不同的场景带来了新的挑战,尤其是在数据摄取方面,现有技术难以处理如此大规模的时序数据,导致加载速度低且时效性差。为应对这些挑战,本文提出了 MatrixGate,一种面向海量时序数据的新型高效数据摄取方法。MatrixGate 基于其独特的摄取策略,实现了单实例和多实例并行过程。首先,MatrixGate 使用策略来调整与 segments 同步的 slots 以摄取数据,从而消除了启动事务的开销并提高了效率。其次,多协程负责传输数据,可显著提高并行度。第三,使用无锁队列实现直接数据传输,无需磁盘存储或驻留在主实例中。在多个数据集上的实验结果表明,MatrixGate 在加载速度上比最先进的方法快 3 到 100 倍,并将查询延迟降低了约 80%。此外,MatrixGate 在分布式架构下可高效横向扩展,实现了 86% 的可扩展性。

英文摘要:

Recent years have seen massive time-series data generated in many areas. This different scenario brings new challenges, particularly in terms of data ingestion, where existing technologies struggle to handle such massive time-series data, leading to low loading speed and poor timeliness. To address these challenges, this paper presents MatrixGate, a new and efficient data ingestion approach for massive time-series data. MatrixGate implements both single-instance and multi-instance parallel procedures, which is based on its unique ingestion strategies. First, MatrixGate uses policies to tune the slots that are synchronized with segments to ingest data, which eliminates the cost of starting transactions and enhance the efficiency. Second, multi-coroutines are responsible for transfer data, which can increase the degree of parallelism significantly. Third, lock-free queues are used to enable direct data transfer without the need for disk storage or lodging in the master instance. Experiment results on multiple datasets show that MatrixGate outperforms state-of-the-art methods by 3 to 100 times in loading speed, and cuts down about 80% query latency. Furthermore, MatrixGate scales out efficiently under distributed architecture, achieving scalability of 86%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑