arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.11432cs.DC
收藏

Octopus:面向分布式科学计算的混合事件驱动架构实践经验

Octopus: Experiences with a Hybrid Event-Driven Architecture for Distributed Scientific Computing

Haochen Pan, Ryan Chard, Sicheng Zhou, Alok Kamatar, Rafael Vescovi, Valérie Hayot-Sasson, André Bauer, Maxime Gonthier, Kyle Chard, Ian Foster

AI总结:

本文提出 Octopus 混合云到边缘事件结构,连接分布式科学计算中的事件生产者与消费者,并通过高吞吐、高可用触发器和细粒度访问控制满足多种科研自动化场景需求。

摘要:

科学研究越来越依赖分布式计算资源、存储系统、网络和仪器,范围从高性能计算和云系统到边缘设备。事件驱动架构(EDA)通过支持对来自众多来源的事件进行组织、通信、处理、可靠性和安全保障,使面向分布式研究基础设施的应用受益。为了支持科学 EDA 的发展,我们引入了 Octopus,这是一种混合的云到边缘事件结构,旨在将许多本地事件生产者和消费者与云托管的代理连接起来。Octopus 可以按需扩展,允许部署高可用的触发器以进行自动事件处理,并实施细粒度的访问控制。我们在自驱动实验室、科学数据自动化、在线任务调度、流行病建模和动态工作流管理用例中确定了需求,并展示了证明 Octopus 满足这些需求能力的结果。Octopus 支持分布式客户端以分别超过每秒 420 万个和 960 万个事件的速率生产和消费事件。

英文摘要:

Scientific research increasingly relies on distributed computational resources, storage systems, networks, and instruments, ranging from HPC and cloud systems to edge devices. Event-driven architecture (EDA) benefits applications targeting distributed research infrastructures by enabling the organization, communication, processing, reliability, and security of events generated from many sources. To support the development of scientific EDA, we introduce Octopus, a hybrid, cloud-to-edge event fabric designed to link many local event producers and consumers with cloud-hosted brokers. Octopus can be scaled to meet demand, permits the deployment of highly available Triggers for automatic event processing, and enforces fine-grained access control. We identify requirements in self-driving laboratories, scientific data automation, online task scheduling, epidemic modeling, and dynamic workflow management use cases, and present results demonstrating Octopus' ability to meet those requirements. Octopus supports producing and consuming events at a rate of over 4.2 M and 9.6 M events per second, respectively, from distributed clients.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑