arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1208.4168cs.DB
收藏

M3R:提高内存中Hadoop作业的性能

M3R: Increased performance for in-memory Hadoop jobs

Avraham Shinnar, David Cunningham, Benjamin Herta, Vijay Saraswat

摘要:

Main Memory Map Reduce(M3 R)是Hadoop Map Reduce(HMR)API的新实现,旨在对高平均故障时间集群进行在线分析。它不支持弹性,只支持那些可以放入集群内存的工作负载。作为回报,它可以运行HMR作业不变-包括由编译器为更高级的语言(如Pig,Jaql和SystemML)和交互式前端(如IBM BigSheets)生成的作业-同时在多个工作负载上提供比Hadoop引擎更好的性能(例如,稀疏矩阵向量乘法的某些输入大小为45倍)。M3 R还支持对HMR API的扩展,这可以使Map Reduce作业在M3 R引擎上运行得更快,同时不会影响其在Hadoop引擎下的性能。

英文摘要:

Main Memory Map Reduce (M3R) is a new implementation of the Hadoop Map Reduce (HMR) API targeted at online analytics on high mean-time-to-failure clusters. It does not support resilience, and supports only those workloads which can fit into cluster memory. In return, it can run HMR jobs unchanged -- including jobs produced by compilers for higher-level languages such as Pig, Jaql, and SystemML and interactive front-ends like IBM BigSheets -- while providing significantly better performance than the Hadoop engine on several workloads (e.g. 45x on some input sizes for sparse matrix vector multiply). M3R also supports extensions to the HMR API which can enable Map Reduce jobs to run faster on the M3R engine, while not affecting their performance under the Hadoop engine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑