使用 Dask 和 ProxyStore 加速 Python 应用
Accelerating Python Applications with Dask and ProxyStore
AI总结:
本文研究将中间件 ProxyStore 与分布式计算库 Dask Distributed 集成,通过高级引用传递范式优化数据流管理,解决 Dask 在数据密集型工作流中的扩展性限制,并在基准测试和实际应用中实现性能提升。
摘要:
应用程序越来越多地被编写为由执行框架支撑的动态工作流,该框架负责管理跨分布式硬件的异步计算。然而,执行框架通常为数据流管理提供一刀切的解决方案,这可能限制性能和可扩展性。ProxyStore 是一种通过高级引用传递范式优化数据流的中间件层,已被证明是解决这些限制的有效机制。在此,我们研究将 ProxyStore 与 Dask Distributed(Python 中最流行的分布式计算库之一)集成,旨在支持可扩展且可移植的科学工作流。Dask 提供了一个易于使用且灵活的框架,但在扩展某些数据密集型工作流方面优化不足。我们调查了这些限制,详细介绍了为分布式应用开发稳健解决方案所需的技术贡献,并在合成基准测试和实际应用中展示了性能提升。
英文摘要:
Applications are increasingly written as dynamic workflows underpinned by an execution framework that manages asynchronous computations across distributed hardware. However, execution frameworks typically offer one-size-fits-all solutions for data flow management, which can restrict performance and scalability. ProxyStore, a middleware layer that optimizes data flow via an advanced pass-by-reference paradigm, has shown to be an effective mechanism for addressing these limitations. Here, we investigate integrating ProxyStore with Dask Distributed, one of the most popular libraries for distributed computing in Python, with the goal of supporting scalable and portable scientific workflows. Dask provides an easy-to-use and flexible framework, but is less optimized for scaling certain data-intensive workflows. We investigate these limitations and detail the technical contributions necessary to develop a robust solution for distributed applications and demonstrate improved performance on synthetic benchmarks and real applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。