The data engineering and data science community has embraced the idea of using Python & R dataframes for regular applications. Driven by the big data revolution and artificial intelligence, these applications are now essential in order to process terabytes of data. They can easily exceed the capabilities of a single machine, but also demand significant developer time & effort. Therefore it is essential to design scalable dataframe solutions. There have been multiple attempts to tackle this problem, the most notable being the dataframe systems developed using distributed computing environments such as Dask and Ray. Even though Dask/Ray distributed computing features look very promising, we perceive that the Dask Dataframes/Ray Datasets still have room for optimization. In this paper, we present CylonFlow, an alternative distributed dataframe execution methodology that enables state-of-the-art performance and scalability on the same Dask/Ray infrastructure (thereby supercharging them!). To achieve this, we integrate a high performance dataframe system Cylon, which was originally based on an entirely different execution paradigm, into Dask and Ray. Our experiments show that on a pipeline of dataframe operators, CylonFlow achieves 30x more distributed performance than Dask Dataframes. Interestingly, it also enables superior sequential performance due to the native C++ execution of Cylon. We believe the success of Cylon & CylonFlow extends beyond the data engineering domain, and can be used to consolidate high performance computing and distributed computing ecosystems.
Supercharging the APGAS Programming Model with Relocatable Distributed Collections
在这篇文章中，我们将介绍我们的可重定位分布式集合库。在AGPAS for Java库的基础上，我们提供了许多有用的节点内并行模式以及通过明确识别的方法支持分布式计算所需的功能。特别地，通过集成的重定位系统来支持分布式集合的条目在进程之间的传输。这启用了动态负载平衡功能，使程序能够适应不均匀或不断变化的集群性能。我们开发的系统使得通过高级抽象动态控制分布式程序的分布和数据流成为可能。因此，使用我们库的程序员可以通过一致的API编写复杂的分布式程序，将计算和通信阶段结合起来。我们评估我们的库的性能对两个程序从著名的Java基准套件，展示了优越的可编程性，并获得更好的性能在一个基准和合理的开销上的第二个。最后，我们将在一个更复杂的应用程序上演示负载平衡的易用性和好处，该应用程序广泛使用我们库的各种功能。
Federated Learning in Chemical Engineering: A Tutorial on a Framework for Privacy-Preserving Collaboration Across Distributed Data Sources
SVKM's Dwarkadas J. Sanghvi College of Engineering（SVKM's 德瓦卡达斯·J.桑格维工程学院）；Davidson School of Chemical Engineering, Purdue University（普渡大学戴维森化学工程学院）；Federal University of Rio de Janeiro（里约热内卢联邦大学）
