arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.16486cs.DC
收藏

ColonyOS——面向异构平台分布式计算的元操作系统

ColonyOS -- A Meta-Operating System for Distributed Computing Across Heterogeneous Platform

Johan Kristiansson

AI总结:

本文提出开源元操作系统ColonyOS,通过分布式执行器与零信任安全协议整合IoT、边缘、云和HPC等异构平台,实现跨平台AI工作流与超分布式应用的无缝运行。

摘要:

本文提出了ColonyOS,一个开源的元操作系统,旨在改善包括IoT、边缘、云和HPC在内的多种计算平台的集成与利用。作为一个覆盖层运行,ColonyOS可以与广泛的计算环境对接,促进所谓计算连续体的创建。这使得开发能够跨平台运行的AI工作流和应用成为可能。ColonyOS的核心由分布式执行器组成,这些执行器基于分布式微服务架构与各种底层平台集成。这些执行器共同形成一个colony,作为统一的计算单元。为了实现各种平台的安全集成,每个colony被精确配置所需的资源,所有通信都限制在由严格零信任安全协议治理的colony内。与ColonyOS的交互通过提交计算任务的功能元描述(称为function specifications)完成。这些描述被发送到Colonies服务器,该服务器作为应用与执行器之间的中介。分配后,执行器解释元描述并将其转换为可执行格式,例如Kubernetes部署描述、Slurm脚本或执行器内的直接函数调用。此外,内置的元文件系统允许将数据同步指令包含在元描述中,实现跨平台的无缝数据管理。最终,ColonyOS为开发可在计算连续体中无缝运行的超分布式应用和工作流铺平了道路。本文描述了ColonyOS的设计原则和实现细节。

英文摘要:

This paper presents ColonyOS, an open-source meta-operating system designed to improve integration and utilization of diverse computing platforms, including IoT, edge, cloud, and HPC. Operating as an overlay, ColonyOS can interface with a wide range of computing environments, fostering creation of so-called compute continuums. This makes it possible to develop AI workflows and applications that can operate across platforms. At its core, ColonyOS consists of distributed executors that integrate with various underlying platforms based on a distributed microservice architecture. These executors collectively form a colony, serving as a unified computing unit. To enable secure integration of various platforms, each colony is provisioned with precisely the resources needed, and all communication is confined within the colony governed by a strict zero-trust security protocol. Interaction with ColonyOS is done by submitting functional meta-descriptions of computational tasks, called function specifications. These are sent to a Colonies server, which acts as intermediary between applications and the executors. Upon assignment, an executor interprets the meta-description and translates it into an executable format, e.g. a Kubernetes deployment description, a Slurm script, or a direct function call within the executor. Furthermore, a built-in meta-file system enables data synchronization directives to be included in meta-descriptions, enabling seamless data management across platforms. Ultimately, ColonyOS paves the way for development of hyper-distributed applications and workflows, which can seamlessly operate in a computing continuum. The paper describes design principles and implementation details of ColonyOS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑