arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.00411cs.DC
收藏

科学工作流的低层 I/O 监控

Low-level I/O Monitoring for Scientific Workflows

Joel Witzke, Ansgar Lößer, Vasilis Bountris, Florian Schintke, Björn Scheuermann

AI总结:

本文研究科学工作流在分布式环境中的低层 I/O 监控问题,提出通过关联低层追踪元数据与 Kubernetes 任务日志信息,在逻辑任务层面分类 I/O 行为,从而识别最值得优化的低效任务。

摘要:

虽然借助适当的工具可以在低层实现详细的资源使用监控,但将这种使用情况与应用层中最初实际引发资源使用的高层抽象关联起来,会带来许多挑战。假设一个大规模科学数据分析工作流运行在分布式执行环境(如计算集群或云环境)中,我们希望分析其 I/O 行为,以发现并缓解潜在瓶颈。工作流的不同任务可以被分配到任意计算节点,甚至可能共享同一计算节点。因此,本地观测到的资源使用并不能直接与各个工作流任务关联。通过获取相关节点的资源使用画像,我们试图将追踪数据与工作流及其各个任务关联起来。为了实现这一目标,我们选择与低层追踪相关联的适当元数据集,从而能够将其与从工作流执行日志文件以及使用 Kubernetes 及其容器管理进行任务编排的作业管理中获得的高层任务信息关联起来。确保正确的信息链,可以在逻辑任务层面分类观测到的 I/O,并可能揭示科学工作流中代价最高或效率最低、因而最值得优化的任务。

英文摘要:

While detailed resource usage monitoring is possible on the low-level using proper tools, associating such usage with higher-level abstractions in the application layer that actually cause the resource usage in the first place presents a number of challenges. Suppose a large-scale scientific data analysis workflow is run using a distributed execution environment such as a compute cluster or cloud environment and we want to analyze the I/O behaviour of it to find and alleviate potential bottlenecks. Different tasks of the workflow can be assigned to arbitrary compute nodes and may even share the same compute nodes. Thus, locally observed resource usage is not directly associated with the individual workflow tasks. By acquiring resource usage profiles of the involved nodes, we seek to correlate the trace data to the workflow and its individual tasks. To accomplish that, we select the proper set of metadata associated with low-level traces that let us associate them with higher-level task information obtained from log files of the workflow execution as well as the job management using a task orchestrator such as Kubernetes with its container management. Ensuring a proper information chain allows the classification of observed I/O on a logical task level and may reveal the most costly or inefficient tasks of a scientific workflow that are most promising for optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑