arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1602.03770cs.DC
收藏

并行流处理引擎中的集成动态重构

Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine

Kasper Grud Skat Madsen, Yongluan Zhou, Jianneng Cao

摘要:

负载均衡、操作实例配置和水平扩展是并行流处理引擎中的关键问题,分别可以实现低数据处理延迟、优化集群利用率和最小化通信开销。在以前的工作中,这些问题通常是单独和独立处理的。我们认为,这些问题是紧密耦合的意义上说,他们都需要在运行时确定工作负载的分配和迁移计算状态。单独优化它们将导致次优解决方案。因此,在本文中,我们研究如何将这三个问题建模为一个集成的优化问题。特别是,我们首先考虑工作量分配的通信成本的影响不大,并建模的负载平衡问题作为一个混合线性规划。之后,我们提出了一个扩展的解决方案称为ALBIC,它支持一般的作业。我们实现的Apache风暴,一个开源的并行流处理引擎的顶部所提出的技术。在合成和真实数据集上的广泛实验结果表明,我们的技术明显优于现有的方法。

英文摘要:

Load balancing, operator instance collocations and horizontal scaling are critical issues in Parallel Stream Processing Engines to achieve low data processing latency, optimized cluster utilization and minimized communication cost respectively. In previous work, these issues are typically tackled separately and independently. We argue that these problems are tightly coupled in the sense that they all need to determine the allocations of workloads and migrate computational states at runtime. Optimizing them independently would result in suboptimal solutions. Therefore, in this paper, we investigate how these three issues can be modeled as one integrated optimization problem. In particular, we first consider jobs where workload allocations have little effect on the communication cost, and model the problem of load balance as a Mixed-Integer Linear Program. Afterwards, we present an extended solution called ALBIC, which support general jobs. We implement the proposed techniques on top of Apache Storm, an open-source Parallel Stream Processing Engine. The extensive experimental results over both synthetic and real datasets show that our techniques clearly outperform existing approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑