在多云环境中表征BigBench查询、Hive和Spark
Characterizing BigBench queries, Hive, and Spark in multi-cloud environments
摘要：
BigBench是用于对大数据系统进行基准测试和测试的新标准（TPCx-BB）。TPCx-BB规范描述了几个业务用例--查询--需要广泛组合数据提取技术，包括SQL、Map/Reduce（M/R）、用户代码（UDF）和机器学习来实现它们。然而，目前，对每项查询的不同所需资源和预期性能没有广泛的了解，而对更既定的基准则是如此。与此同时，云计算提供商目前提供方便的按需管理的大数据集群（PaaS），采用按需付费模式。在PaaS中，Hive和Spark等分析引擎可以随时使用，具有通用配置和升级管理。该研究描述了BigBench查询以及Spark和Hive版本在云中的开箱即用性能。与此同时，从Azure HDinsight、Amazon Web Services EMR和Google Cloud Dataproc的可靠性、数据可扩展性（1GB到10 TB）、版本和设置方面比较流行的PaaS产品。查询特征突出了Hive和Spark框架的相似性和差异，以及根据CPU，内存和I/O，哪些查询消耗最多的资源。可扩展性结果显示，随着数据规模的增长，大多数云提供商都需要进行配置调优，特别是在Sparks内存使用情况下。这些结果可以帮助从业者通过选择强调每个类别的查询子集来快速测试系统。与此同时，结果显示了Hive和Spark的比较情况，以及在PaaS中可以预期的性能。
英文摘要：
BigBench is the new standard (TPCx-BB) for benchmarking and testing Big Data systems. The TPCx-BB specification describes several business use cases -- queries -- which require a broad combination of data extraction techniques including SQL, Map/Reduce (M/R), user code (UDF), and Machine Learning to fulfill them. However, currently, there is no widespread knowledge of the different resource requirements and expected performance of each query, as is the case to more established benchmarks. At the same time, cloud providers currently offer convenient on-demand managed big data clusters (PaaS) with a pay-as-you-go model. In PaaS, analytical engines such as Hive and Spark come ready to use, with a general-purpose configuration and upgrade management. The study characterizes both the BigBench queries and the out-of-the-box performance of Spark and Hive versions in the cloud. At the same time, comparing popular PaaS offerings in terms of reliability, data scalability (1GB to 10TB), versions, and settings from Azure HDinsight, Amazon Web Services EMR, and Google Cloud Dataproc. The query characterization highlights the similarities and differences in Hive an Spark frameworks, and which queries are the most resource consuming according to CPU, memory, and I/O. Scalability results show how there is a need for configuration tuning in most cloud providers as data scale grows, especially with Sparks memory usage. These results can help practitioners to quickly test systems by picking a subset of the queries which stresses each of the categories. At the same time, results show how Hive and Spark compare and what performance can be expected of each in PaaS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
