DataMPI的性能优势:使用BigDataBench的案例研究
Performance Benefits of DataMPI: A Case Study with BigDataBench
摘要:
Apache Hadoop和Spark在大数据处理和分析方面越来越突出。两者都在互联网公司中广泛部署。另一方面,高性能数据分析需求促使学术界和工业界采用HPC中最先进的技术来解决大数据问题。最近,我们提出了一个基于键值对的通信库DataMPI,它扩展了MPI以支持Hadoop/Spark之类的大数据计算作业。本文使用BigDataBench这一大数据基准测试套件,对Hadoop、Spark和DataMPI的性能和资源利用率特征进行了全面的研究。从我们的实验中,我们观察到DataMPI的作业执行时间分别比Hadoop和Spark的作业执行时间提高了55%和39%。大部分好处来自DataMPI中的高效通信机制。我们还注意到,DataMPI的资源(CPU,内存,磁盘和网络I/O)利用率也比其他两个框架更有效。
英文摘要:
Apache Hadoop and Spark are gaining prominence in Big Data processing and analytics. Both of them are widely deployed on Internet companies. On the other hand, high-performance data analysis requirements are causing academical and industrial communities to adopt state-of-the-art technologies in HPC to solve Big Data problems. Recently, we have proposed a key-value pair based communication library, DataMPI, which is extending MPI to support Hadoop/Spark-like Big Data Computing jobs. In this paper, we use BigDataBench, a Big Data benchmark suite, to do comprehensive studies on performance and resource utilization characterizations of Hadoop, Spark and DataMPI. From our experiments, we observe that the job execution time of DataMPI has up to 55% and 39% speedups compared with those of Hadoop and Spark, respectively. Most of the benefits come from the high-efficiency communication mechanisms in DataMPI. We also notice that the resource (CPU, memory, disk and network I/O) utilizations of DataMPI are also more efficient than those of the other two frameworks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。