arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1503.06600cs.DC
收藏

通过Hive和Pig探索高尺度云的非均匀性和动力性

Exploring Non-Homogeneity and Dynamicity of High Scale Cloud through Hive and Pig

Kashish Ara Shakil, Mansaf Alam, Shuchi Sethi

摘要:

云计算处理所有级别的异构性和动态性,因此需要管理这种环境中的资源并适当地分配它们。资源规划和调度需要正确理解资源的到达模式和调度。对工作负载的研究有助于正确理解其相关环境。Google在2014年11月发布了其最新版本的集群跟踪,跟踪版本2.1。该跟踪由跨越70万个作业的大约29天的单元格信息组成。本文对这一聚类轨迹进行了统计分析。由于跟踪的大小非常大,因此使用了Hive,Hive是用于查询和分析大数据的基于Hadoop分布式文件系统(HDFS)的平台。Hive是通过Beeswax接口访问的。数据通过HCcatalog导入HDFS。除了Hive之外,还使用了Pig,这是一种脚本语言,在Hadoop之上提供抽象。据我们所知,我们采用的分析方法是新颖的,有助于获得一些有用的见解。本文使用K-means++聚类对工件和到达时间进行聚类,然后分析工件到达时间的分布,发现工件到达时间分布为威布尔分布,而资源使用率接近于zip-f分布,过程运行时间分布为重尾分布。

英文摘要:

Cloud computing deals with heterogeneity and dynamicity at all levels and therefore there is a need to manage resources in such an environment and properly allocate them. Resource planning and scheduling requires a proper understanding of arrival patterns and scheduling of resources. Study of workloads can aid in proper understanding of their associated environment. Google has released its latest version of cluster trace, trace version 2.1 in November 2014.The trace consists of cell information of about 29 days spanning across 700k jobs. This paper deals with statistical analysis of this cluster trace. Since the size of trace is very large, Hive which is a Hadoop distributed file system (HDFS) based platform for querying and analysis of Big data, has been used. Hive was accessed through its Beeswax interface. The data was imported into HDFS through HCatalog. Apart from Hive, Pig which is a scripting language and provides abstraction on top of Hadoop was used. To the best of our knowledge the analytical method adopted by us is novel and has helped in gaining several useful insights. Clustering of jobs and arrival time has been done in this paper using K-means++ clustering followed by analysis of distribution of arrival time of jobs which revealed weibull distribution while resource usage was close to zip-f like distribution and process runtimes revealed heavy tailed distribution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑