arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.05072cs.DC
收藏

Exoshuffle:一种可扩展的 Shuffle 架构

Exoshuffle: An Extensible Shuffle Architecture

Frank Sifei Luan, Stephanie Wang, Samyukta Yagati, Sean Kim, Kenneth Lien, Isaac Ong, Tony Hong, SangBin Cho, Eric Liang, Ion Stoica

AI总结:

本文提出构建于 Ray 之上的可扩展 shuffle 库 Exoshuffle,通过解耦控制平面与数据平面,以更少代码复现既有优化,并取得具竞争力的性能与 CloudSort 成本纪录。

摘要:

Shuffle 是分布式数据处理中开销最高的通信原语之一,且难以扩展。以往工作通过构建单体式 shuffle 系统来解决 shuffle 的可扩展性挑战。这些系统开发成本高昂,并且与仅提供 SQL 等高层 API 的批处理框架紧密集成。ML 训练等新应用需要更高的灵活性以及与 shuffle 更细粒度的互操作性,却往往无法利用现有的 shuffle 优化。我们提出一种可扩展的 shuffle 架构。我们介绍了 Exoshuffle,这是一个用于分布式 shuffle 的库,可提供具有竞争力的性能和可扩展性,同时比单体式 shuffle 系统具备更高的灵活性。我们设计了一种在不牺牲性能的情况下将 shuffle 控制平面与数据平面解耦的架构。我们在面向数据和 ML 应用的分布式 futures 系统 Ray 上构建 Exoshuffle,并证明我们能够:(1)以少一个数量级的代码,将以往的 shuffle 优化重写为应用级库;(2)实现可与单体式 shuffle 系统竞争的 shuffle 性能和可扩展性,并打破 CloudSort 纪录,成为全球最具成本效益的排序系统;(3)使 ML 训练等新应用能够轻松利用可扩展的 shuffle。

英文摘要:

Shuffle is one of the most expensive communication primitives in distributed data processing and is difficult to scale. Prior work addresses the scalability challenges of shuffle by building monolithic shuffle systems. These systems are costly to develop, and they are tightly integrated with batch processing frameworks that offer only high-level APIs such as SQL. New applications, such as ML training, require more flexibility and finer-grained interoperability with shuffle. They are often unable to leverage existing shuffle optimizations. We propose an extensible shuffle architecture. We present Exoshuffle, a library for distributed shuffle that offers competitive performance and scalability as well as greater flexibility than monolithic shuffle systems. We design an architecture that decouples the shuffle control plane from the data plane without sacrificing performance. We build Exoshuffle on Ray, a distributed futures system for data and ML applications, and demonstrate that we can: (1) rewrite previous shuffle optimizations as application-level libraries with an order of magnitude less code, (2) achieve shuffle performance and scalability competitive with monolithic shuffle systems, and break the CloudSort record as the world's most cost-efficient sorting system, and (3) enable new applications such as ML training to easily leverage scalable shuffle.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑