arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.02531cs.DC
收藏

云和外部资源之间的Kubernetes“桥梁”运营商

A Kubernetes 'Bridge' operator between cloud and external resources

Boris Lublinsky, Elise Jennings, Viktória Spišaková

摘要:

许多科学工作流需要专用的计算资源,包括具有优化软件的HPC集群、量子资源和专用硬件集群系统(例如Ray)。与此同时,今天的许多科学工作流都建立在Kubernetes上,利用对工作流和支持工具的日益增长的支持。为了满足在云和专用计算资源上支持工作流的日益增长的需求,我们提出了Bridge Operator,这是Kubernetes中容器编排的软件扩展,可以方便地提交和监控具有自己的集群资源管理器(SLURM,LSF,量子服务和Ray)的外部系统上的长时间运行进程。Bridge Operator由一个自定义Kubernetes控制器组成,它使用Kubernetes自定义资源定义来管理应用程序。我们提出了用于管理云容器编排和外部资源工作负载管理器接口的控制器逻辑、用于向外部资源提交HTTP/HTTPS请求的资源定义以及与外部资源管理器通信以提交和管理作业执行的控制器pod。该实现允许我们在Kubernetes pod中镜像外部资源,这允许操作员使用这些pod作为代理来控制外部系统。该实现与资源管理器的选择无关,但假设系统公开了HTTP/HTTPS API以进行控制/管理。Bridge Operator自动化了操作员在黑盒外部资源上运行作业的角色,作为云上复杂混合工作流的一部分。

英文摘要:

Many scientific workflows require dedicated compute resources, including HPC clusters with optimized software, quantum resources, and dedicated hardware cluster systems like Ray, for example. At the same time, many scientific workflows today are built on Kubernetes leveraging growing support for workflow and support tools. To address the growing demand to support workflows on both cloud and dedicated compute resources we present the Bridge Operator, a software extension for container orchestration in Kubernetes which facilitates the submission and monitoring of long running processes on external systems which have their own cluster resources manager (SLURM, LSF, quantum services and Ray). The Bridge Operator consists of a custom Kubernetes controller that employs a Kubernetes Custom Resource Definition to manage applications. We present controller logic to manage the cloud container orchestration and external resource workload manager interface, a resource definition to submit HTTP/HTTPS requests to the external resource, and a controller pod communicating with the external resource manager to submit and manage job execution. The implementation allows us to mirror the external resource in Kubernetes pods, which allows the operator to use these pods as proxies to control the external system. The implementation is agnostic to the choice of resource manager but assumes the system exposes a HTTP/HTTPS API for its control/management. The Bridge Operator automates the role of a human operator running jobs on a black box external resource as part of a complex hybrid workflow on the Cloud.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑