工作流引擎应如何与资源管理器对话：通用工作流调度接口提案
How Workflow Engines Should Talk to Resource Managers: A Proposal for a Common Workflow Scheduling Interface
AI总结：
针对 SWMS 与资源管理器职责重叠、接口不统一导致的可移植性差和次优调度问题，论文提出通用 REST 调度接口，并通过 Nextflow/Kubernetes 实现验证其可显著降低工作流完工时间。
摘要：
科学工作流管理系统（SWMS）与资源管理器共同确保任务被调度到已供给的资源上，从而遵守所有依赖关系，并实现某种优化目标，例如最小化完工时间。然而在实践中，SWMS 与资源管理器之间并没有清晰的调度职责划分，因为其不同组件之间不存在公认的关注点分离。这会带来两个后果。第一，缺乏用于在 SWMS 与资源管理器之间交换调度信息的标准化 API，阻碍了可移植性。当某个组件需要被另一个组件替换时（例如在同一资源管理器上将一个 SWMS 替换为另一个 SWMS），这会导致高昂的适配成本。第二，由于功能重叠，当前部署实际上往往存在两个调度器，二者都在信息不完整的情况下作出部分调度决策，从而导致工作流调度结果次优。本文提出一种位于 SWMS 与资源管理器之间的简单 REST 接口，它允许任何 SWMS 将动态工作流信息传递给资源管理器，从而支持在信息最充分的情况下作出调度决策。我们提供了该 API 的一个实现示例，其中使用 Nextflow 作为 SWMS，并使用 Kubernetes 作为资源管理器。我们针对九个真实世界工作流开展的实验表明，与标准 Nextflow/Kubernetes 配置相比，该策略最多可将完工时间降低 25.1%，平均降低 10.8%。此外，更广泛地实现该 API 将使代码库更精简、工作流系统组件交换更简单，并为实现新的调度算法提供统一位置。
英文摘要：
Scientific workflow management systems (SWMSs) and resource managers together ensure that tasks are scheduled on provisioned resources so that all dependencies are obeyed, and some optimization goal, such as makespan minimization, is achieved. In practice, however, there is no clear separation of scheduling responsibilities between an SWMS and a resource manager because there exists no agreed-upon separation of concerns between their different components. This has two consequences. First, the lack of a standardized API to exchange scheduling information between SWMSs and resource managers hinders portability. It incurs costly adaptations when a component should be replaced by a different one (e.g., an SWMS with another SWMS on the same resource manager). Second, due to overlapping functionalities, current installations often actually have two schedulers, both making partial scheduling decisions under incomplete information, leading to suboptimal workflow scheduling. In this paper, we propose a simple REST interface between SWMSs and resource managers, which allows any SWMS to pass dynamic workflow information to a resource manager, enabling maximally informed scheduling decisions. We provide an implementation of this API as an example, using Nextflow as an SWMS and Kubernetes as a resource manager. Our experiments with nine real-world workflows show that this strategy reduces makespan by up to 25.1% and 10.8% on average compared to the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread implementation of this API would enable leaner code bases, a simpler exchange of components of workflow systems, and a unified place to implement new scheduling algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
