arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.07475cs.DC
收藏

kiwiPy:用于大数据和计算科学工作流的强大、高容量消息传递

kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows

Martin Uhrin, Sebastiaan P. Huber

摘要:

在这项工作中,我们提出了kiwiPy,一个Python库,旨在支持高吞吐量,大数据,应用程序的强大的基于消息的通信,同时足够通用,在需要以可预测的方式传递大量消息的情况下非常有用。KiwiPy依赖于RabbitMQ协议,这是一种行业标准的消息代理,同时提供了一个简单直观的界面,可用于多线程和基于协同程序的应用程序。为了演示kiwiPy的一些功能,我们给出了来自AiiDA的示例,AiiDA是一个高吞吐量的仿真平台,其中kiwiPy被用作工作流引擎的关键组件。

英文摘要:

In this work we present kiwiPy, a Python library designed to support robust message based communication for high-throughput, big-data, applications while being general enough to be useful wherever high-volumes of messages need to be communicated in a predictable manner. KiwiPy relies on the RabbitMQ protocol, an industry standard message broker, while providing a simple and intuitive interface that can be used in both multithreaded and coroutine based applications. To demonstrate some of kiwiPy's functionality we give examples from AiiDA, a high-throughput simulation platform, where kiwiPy is used as a key component of the workflow engine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑