扇区与球体:大规模分布式数据的简化存储与处理
Sector and Sphere: Towards Simplified Storage and Processing of Large Scale Distributed Data
摘要:
云计算已经证明,只要给出正确的编程模型和基础设施,就可以在商品集群上处理非常大的数据集。在本文中,我们描述了Sector存储云和Sphere计算云的设计和实现。与现有的存储和计算云相比,Sector不仅可以在数据中心内管理数据,还可以跨地理分布的数据中心管理数据。同样,Sphere计算云支持数据中心内和跨数据中心的用户定义函数(UDF)。作为一种特殊情况,MapReduce风格的编程可以在Sphere中通过使用Map UDF和Reduce UDF来实现。我们描述了一些实验研究比较部门/球体和Hadoop使用Terasort基准。在这些研究中,Sector的速度大约是Hadoop的两倍。Sector/Sphere是开源的。
英文摘要:
Cloud computing has demonstrated that processing very large datasets over commodity clusters can be done simply given the right programming model and infrastructure. In this paper, we describe the design and implementation of the Sector storage cloud and the Sphere compute cloud. In contrast to existing storage and compute clouds, Sector can manage data not only within a data center, but also across geographically distributed data centers. Similarly, the Sphere compute cloud supports User Defined Functions (UDF) over data both within a data center and across data centers. As a special case, MapReduce style programming can be implemented in Sphere by using a Map UDF followed by a Reduce UDF. We describe some experimental studies comparing Sector/Sphere and Hadoop using the Terasort Benchmark. In these studies, Sector is about twice as fast as Hadoop. Sector/Sphere is open source.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。