数据流处理系统抽象层的定量影响评估
Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems
摘要:
随着对处理不断增长的数据量的需求,已经开发了各种新的数据流处理框架。将实现从一个这样的系统移动到另一个这样的系统,例如,出于性能原因,需要使现有应用程序适应新接口。Apache Beam通过提供一个抽象层来解决这些高替代成本,该抽象层允许在任何支持的流框架上执行程序。在本文中,我们提出了一种新的基准架构,用于比较使用Apache Beam对三种流媒体框架的性能影响:Apache Spark Streaming,Apache Flink和Apache Apex。我们发现,在调查的系统中使用Apache Beam进行应用程序开发时,性能会受到显着影响。总的来说,使用Apache Beam进行检查流应用程序会导致查询执行时间的差异很大,与没有抽象层的查询相比,查询执行时间的延迟高达58倍。所有开发的基准工件都是公开的,以确保可重复的结果。
英文摘要:
With the demand to process ever-growing data volumes, a variety of new data stream processing frameworks have been developed. Moving an implementation from one such system to another, e.g., for performance reasons, requires adapting existing applications to new interfaces. Apache Beam addresses these high substitution costs by providing an abstraction layer that enables executing programs on any of the supported streaming frameworks. In this paper, we present a novel benchmark architecture for comparing the performance impact of using Apache Beam on three streaming frameworks: Apache Spark Streaming, Apache Flink, and Apache Apex. We find significant performance penalties when using Apache Beam for application development in the surveyed systems. Overall, usage of Apache Beam for the examined streaming applications caused a high variance of query execution times with a slowdown of up to a factor of 58 compared to queries developed without the abstraction layer. All developed benchmark artifacts are publicly available to ensure reproducible results.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。