旋转快速迭代数据流
Spinning Fast Iterative Data Flows
摘要:
并行数据流系统是大多数大数据分析管道的核心部分。然而,许多分析和机器学习算法的迭代性质对于当前系统仍然是一个挑战。虽然某些类型的批量迭代算法由新颖的Bulllow框架支持,但是这些系统不能利用存在于许多算法(诸如图算法)中的计算依赖性。结果,这些算法被低效地执行,并导致基于其他范例的专用系统,例如消息传递或共享内存。我们提出了一种方法来集成增量迭代,一种形式的工作集迭代,与并行并行的并行处理。在展示了如何将批量迭代集成到一个Cocklow系统及其优化器中之后,我们提出了一个对增量迭代编程模型的扩展。该扩展解释了在递归中缺乏可变状态,并允许利用许多迭代算法中固有的稀疏计算依赖关系。一个原型实现的评估表明,这些方面导致了两个数量级的加速算法运行时,当利用。在我们的实验中,改进后的系统是非常有竞争力的专业系统,同时保持一个透明的和统一的抽象的抽象。
英文摘要:
Parallel dataflow systems are a central part of most analytic pipelines for big data. The iterative nature of many analysis and machine learning algorithms, however, is still a challenge for current systems. While certain types of bulk iterative algorithms are supported by novel dataflow frameworks, these systems cannot exploit computational dependencies present in many algorithms, such as graph algorithms. As a result, these algorithms are inefficiently executed and have led to specialized systems based on other paradigms, such as message passing or shared memory. We propose a method to integrate incremental iterations, a form of workset iterations, with parallel dataflows. After showing how to integrate bulk iterations into a dataflow system and its optimizer, we present an extension to the programming model for incremental iterations. The extension alleviates for the lack of mutable state in dataflows and allows for exploiting the sparse computational dependencies inherent in many iterative algorithms. The evaluation of a prototypical implementation shows that those aspects lead to up to two orders of magnitude speedup in algorithm runtime, when exploited. In our experiments, the improved dataflow system is highly competitive with specialized systems while maintaining a transparent and unified dataflow abstraction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。