arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.06146cs.DC
收藏

基于并行处理模式的高性能数据框

High Performance Dataframes from Parallel Processing Patterns

Niranda Perera, Supun Kamburugamuve, Chathura Widanage, Vibhatha Abeykoon, Ahmet Uyar, Kaiying Shan, Hasara Maithree, Damitha Lenadora, Thejaka Amila Kanewala, Geoffrey Fox

AI总结:

本文提出基于并行处理模式的分布式数据框框架,并以 Cylon 运行时实现验证其可扩展高性能、API 灵活性与跨硬件可扩展性。

摘要:

如今,数据科学界已将数据框(Dataframes)视为数据表示与操作的事实标准。易用性、大规模算子覆盖以及 R 和 Python 语言的普及极大地推动了这一转变。然而,当今最广泛使用的串行数据框(R、pandas)即使在处理中等规模数据集时也会遇到性能限制。我们相信,通过研究数据框算子的通用分布式模式,仍有很大的改进空间。本文提出了一个框架,为基于这些并行处理模式构建高性能分布式内存并行数据框系统奠定了基础。我们还介绍了 Cylon 作为参考运行时实现。我们展示了该框架如何使 Cylon 实现可扩展的高性能。我们还强调了所提出 API 的灵活性以及该框架在不同硬件上的可扩展性。据我们所知,Cylon 是当今第一个也是唯一一个可用的分布式内存并行数据框系统。

英文摘要:

The data science community today has embraced the concept of Dataframes as the de facto standard for data representation and manipulation. Ease of use, massive operator coverage, and popularization of R and Python languages have heavily influenced this transformation. However, most widely used serial Dataframes today (R, pandas) experience performance limitations even while working on even moderately large data sets. We believe that there is plenty of room for improvement by investigating the generic distributed patterns of dataframe operators. In this paper, we propose a framework that lays the foundation for building high performance distributed-memory parallel dataframe systems based on these parallel processing patterns. We also present Cylon, as a reference runtime implementation. We demonstrate how this framework has enabled Cylon achieving scalable high performance. We also underline the flexibility of the proposed API and the extensibility of the framework on different hardware. To the best of our knowledge, Cylon is the first and only distributed-memory parallel dataframe system available today.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑