arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1509.02256cs.DC
收藏

Apache Spark中的矩阵计算和优化

Matrix Computations and Optimization in Apache Spark

Reza Bosagh Zadeh, Xiangrui Meng, Aaron Staple, Burak Yavuz, Li Pu, Shivaram Venkataraman, Evan Sparks, Alexander Ulanov, Matei Zaharia

摘要:

我们描述了集群编程框架Apache Spark中的矩阵计算。Spark开箱即用,为分布式矩阵和使用这些矩阵的优化例程提供了抽象和实现。当将单节点算法转换为在分布式集群上运行时,我们观察到通常一个简单的想法就足够了:将矩阵操作与向量操作分离,并将矩阵操作运送到集群上运行,同时将向量操作保持在驱动程序本地。在奇异值分解的情况下,通过将这个想法发挥到极致,我们能够利用集群的计算能力,同时运行几十年前为单核编写的代码。另一个例子是我们流行的TFOCS优化包的Spark端口,最初是为MATLAB构建的,它允许求解线性规划以及各种其他凸规划。最后,我们总结了一套全面的基准测试,用于从JVM进行硬件加速矩阵计算,这本身就很有趣,因为许多集群编程框架都使用JVM。本文中描述的贡献已经合并到Apache Spark中,默认情况下可在Spark安装中使用,并由一系列提供进一步服务的公司提供商业支持。

英文摘要:

We describe matrix computations available in the cluster programming framework, Apache Spark. Out of the box, Spark provides abstractions and implementations for distributed matrices and optimization routines using these matrices. When translating single-node algorithms to run on a distributed cluster, we observe that often a simple idea is enough: separating matrix operations from vector operations and shipping the matrix operations to be ran on the cluster, while keeping vector operations local to the driver. In the case of the Singular Value Decomposition, by taking this idea to an extreme, we are able to exploit the computational power of a cluster, while running code written decades ago for a single core. Another example is our Spark port of the popular TFOCS optimization package, originally built for MATLAB, which allows for solving Linear programs as well as a variety of other convex programs. We conclude with a comprehensive set of benchmarks for hardware accelerated matrix computations from the JVM, which is interesting in its own right, as many cluster programming frameworks use the JVM. The contributions described in this paper are already merged into Apache Spark and available on Spark installations by default, and commercially supported by a slew of companies which provide further services.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑