Labyrinth:将命令控制流编译为并行数据流
Labyrinth: Compiling Imperative Control Flow to Parallel Dataflows
摘要:
并行数据流系统已经成为大规模数据分析的标准技术。在诸如机器学习和图形分析等领域中的复杂数据分析程序通常涉及控制流,即,迭代和分支。因此,用于高级分析的系统应该包括高效且易于使用的控制流构造。一种自然的方法是提供类似于主流程序设计语言的命令式控制流结构:while循环、if语句和可变变量,这些变量的值可以在迭代步骤之间改变。然而,当前的并行并行控制流系统通过在每个控制流决策之后启动单独的控制流作业(例如,对于循环的每一步)。这种方法的性能是次优的,因为(a)启动一个低效率的作业会导致调度开销;(b)它会阻止跨迭代步骤的某些优化。在本文中,我们介绍了Labeldth,一种方法来编译使用命令式控制流结构编写的程序到一个单一的并行作业,它执行整个程序,包括所有的迭代步骤。通过这种方式,我们实现了效率和易用性。我们还进行了实验评估,结果表明,Labeldth的数量级较小的每迭代步骤的开销比启动新的工作,也允许跨迭代步骤的显着优化。
英文摘要:
Parallel dataflow systems have become a standard technology for large-scale data analytics. Complex data analysis programs in areas such as machine learning and graph analytics often involve control flow, i.e., iterations and branching. Therefore, systems for advanced analytics should include control flow constructs that are efficient and easy to use. A natural approach is to provide imperative control flow constructs similar to those of mainstream programming languages: while-loops, if-statements, and mutable variables, whose values can change between iteration steps. However, current parallel dataflow systems execute programs written using imperative control flow constructs by launching a separate dataflow job after every control flow decision (e.g., for every step of a loop). The performance of this approach is suboptimal, because (a) launching a dataflow job incurs scheduling overhead; and (b) it prevents certain optimizations across iteration steps. In this paper, we introduce Labyrinth, a method to compile programs written using imperative control flow constructs to a single dataflow job, which executes the whole program, including all iteration steps. This way, we achieve both efficiency and ease of use. We also conduct an experimental evaluation, which shows that Labyrinth has orders of magnitude smaller per-iteration-step overhead than launching new dataflow jobs, and also allows for significant optimizations across iteration steps.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。