arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.10106cs.DC
收藏

ds-array:一种用于大规模机器学习的分布式数据结构

ds-array: A Distributed Data Structure for Large Scale Machine Learning

Javier Álvarez Cid-Fuentes, Pol Álvarez, Salvi Solà, Kuninori Ishii, Rafael K. Morizawa, Rosa M. Badia

摘要:

机器学习已被证明是从许多研究领域的科学数据中提取知识的有用工具,包括天体物理学,基因组学和分子动力学。通常,来自这些研究领域的数据集由于其规模而需要在分布式平台中处理。这可以使用各种可用的分布式机器学习库之一来完成。其中一个库是dislib,这是一个用于Python的分布式机器学习库,专门设计用于在HPC集群上处理大规模数据集,这使得dislib成为分析科学数据的理想候选者。但是,dislib的主要分布式数据结构(称为Dataset)有一些局限性,包括某些操作的性能较差,灵活性和可用性较低。在本文中,我们提出了一种新的分布式数据结构的dislib,称为ds-array,解决了dislib的数据管理的主要局限性。DS-arrays通过公开类似NumPy的API简化了dislib中的分布式数据管理,提供了更大的灵活性,并降低了某些操作的计算复杂性。这导致性能比数据集提高了两个数量级,同时也大大提高了可伸缩性和可用性。

英文摘要:

Machine learning has proved to be a useful tool for extracting knowledge from scientific data in numerous research fields, including astrophysics, genomics, and molecular dynamics. Often, data sets from these research areas need to be processed in distributed platforms due to their magnitude. This can be done using one of the various distributed machine learning libraries available. One of these libraries is dislib, a distributed machine learning library for Python especially designed to process large scale data sets on HPC clusters, which makes dislib an ideal candidate for analyzing scientific data. However, dislib's main distributed data structure, called Dataset, has some limitations, including poor performance in certain operations and low flexibility and usability. In this paper, we propose a novel distributed data structure for dislib, called ds-array, that addresses dislib's main limitations in data management. Ds-arrays simplify distributed data management in dislib by exposing a NumPy-like API, provide more flexibility, and reduce the computational complexity of some operations. This results in performance improvements of up to two orders of magnitude over Datasets, while also greatly improving scalability and usability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑