arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16353cs.DC
收藏

Sizey:科学工作流任务的内存高效执行

Sizey: Memory-Efficient Execution of Scientific Workflow Tasks

Jonathan Bader, Fabian Skalski, Fabian Lehmann, Dominik Scheinert, Jonathan Will, Lauritz Thamsen, Odej Kao

AI总结:

本文提出在线内存预测方法Sizey,通过同时训练多个机器学习模型并动态选择最佳模型来优化科学工作流任务的内存分配,在六个nf-core真实工作流上相比最佳基线中位数减少内存浪费24.68%。

摘要:

随着生物信息学、物理学和遥感等多样化领域中可用数据量的持续增长,科学工作流在可复现数据分析管道的设计与实现中的重要性日益增加。在开发工作流时,必须为工作流中的每种任务类型定义资源需求。通常,任务类型的计算需求差异很大,因为它们只是任意黑盒分析工具的封装器。此外,由于输入不同,同一任务类型的资源消耗也可能有很大差异。由于低估内存资源会导致瓶颈和任务失败,工作流开发者倾向于高估内存资源。然而,内存的过度配置浪费资源并限制集群吞吐量。针对这一问题,我们提出了Sizey,一种新颖的面向工作流任务的在线内存预测方法。在工作流执行过程中,Sizey同时训练多个机器学习模型,然后为每个工作流任务动态选择最佳模型。为了评估模型质量,我们引入了一种基于内存预测准确性和效率的新型资源分配质量(RAQ)评分。Sizey的预测模型在工作流执行期间在线重新训练和重新评估,持续纳入已完成任务的指标。我们使用Sizey原型实现的评估采用了来自流行的nf-core框架的六个真实世界科学工作流的指标,结果显示与各自表现最佳的最先进基线相比,内存浪费随时间的中位数减少了24.68%。

英文摘要:

As the amount of available data continues to grow in fields as diverse as bioinformatics, physics, and remote sensing, the importance of scientific workflows in the design and implementation of reproducible data analysis pipelines increases. When developing workflows, resource requirements must be defined for each type of task in the workflow. Typically, task types vary widely in their computational demands because they are simply wrappers for arbitrary black-box analysis tools. Furthermore, the resource consumption for the same task type can vary considerably as well due to different inputs. Since underestimating memory resources leads to bottlenecks and task failures, workflow developers tend to overestimate memory resources. However, overprovisioning of memory wastes resources and limits cluster throughput. Addressing this problem, we propose Sizey, a novel online memory prediction method for workflow tasks. During workflow execution, Sizey simultaneously trains multiple machine learning models and then dynamically selects the best model for each workflow task. To evaluate the quality of the model, we introduce a novel resource allocation quality (RAQ) score based on memory prediction accuracy and efficiency. Sizey's prediction models are retrained and re-evaluated online during workflow execution, continuously incorporating metrics from completed tasks. Our evaluation with a prototype implementation of Sizey uses metrics from six real-world scientific workflows from the popular nf-core framework and shows a median reduction in memory waste over time of 24.68% compared to the respective best-performing state-of-the-art baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑