增强分布式任务型工作流的迭代性能
Enhancing iteration performance on distributed task-based workflows
AI总结：
本文提出SplIter机制，通过将数据集合透明地分割为逻辑分区并绑定任务，打破块大小与任务粒度的依赖，从而在COMPSs和Dask框架中显著提升分布式任务型工作流的迭代性能。
摘要：
任务型编程模型已被证明是开发分布式环境应用程序的一种稳健且通用的方法。它们提供了自然的编程模式并具有高性能。然而，在该范式下的执行对粒度——即任务的数量和执行长度——非常敏感。粒度通常与数据的块大小相关，而寻找最优块大小面临若干挑战，因为它需要计算环境的内部知识。我们的提议是用一种新机制——我们的SplIter方案——来补充任务型编程模型。其核心在于，SplIter提供了一种透明的方式，将集合分割为多个分区（块的逻辑分组，无需任何传输或数据重排即可获得），随后可对这些分区进行迭代。任务与这些分区相关联，这意味着SplIter打破了块大小与任务粒度之间的依赖关系。评估表明，与基线相比，SplIter能够实现超过一个数量级的性能提升，并且相对于竞争性替代方案，它要么具有竞争力，要么严格更优（取决于应用特性）。我们选择了覆盖多种场景的不同应用；这些应用代表了更广泛的应用和领域集合。对任务型应用源代码所需的改动极小，从而保持了该编程模型的高可编程性。针对所有应用，我们评估了两种不同的最先进任务型框架：COMPSs和Dask，结果表明SplIter可以在不同框架内有效使用。
英文摘要：
Task-based programming models have proven to be a robust and versatile way to approach development of applications for distributed environments. They provide natural programming patterns with high performance. However, execution on this paradigm can be very sensitive to granularity --i.e., the quantity and execution length of tasks. Granularity is often linked with the block size of the data, and finding the optimal block size has several challenges, as it requires inner knowledge of the computing environment. Our proposal is to supplement the task-based programming model with a new mechanism --our SplIter proposal. At its core, the SplIter provides a transparent way to split a collection into partitions (logical groups of blocks, obtained without any transfers nor data rearrangement), which can then be iterated. Tasks are linked to those partitions, which means that SplIter breaks the dependency between block size and task granularity. The evaluation shows that the SplIter is able to achieve performance improvements of over one order of magnitude when compared to the baseline, and it is either competitive or strictly better (depending on application characteristics) to the competitor alternative. We have chosen different applications covering a wide variety of scenarios; those applications are representatives of a broader set of applications and domains. The changes required in the source code of a task-based application are minimal, preserving the high programmability of the programming model. Two different state-of-the-art task-based frameworks have been evaluated for all the applications: COMPSs and Dask, showing that the SplIter can be effectively used within different frameworks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
