Hadoop中具有固有双重复制的代码评估
Evaluation of Codes with Inherent Double Replication for Hadoop
摘要：
在本文中，我们评估的功效，在Hadoop的设置，两个编码方案，都拥有一个固有的双重复制的数据。这两种编码方案分别属于再生和局部再生代码类，这两类代码代表了在分布式环境中有效存储数据的代码设计方面取得的最新进展。与三重复制相比，双重复制可以显著减少存储开销，同时在中等工作负载下提供良好的MapReduce性能。这里评估的两种编码解决方案仅适度增加了双重复制的存储开销，同时提供了类似于三重复制的可靠性级别。从固有的数据复制属性中，人们可能会期望这些代码在执行MapReduce作业时的性能与双重复制相当。然而，这类代码的第二个特征在这里起作用，即在这里分析的两种编码方案下，来自同一编码条带的多个块需要存储在同一节点上。这种属于单个条带的数据集中会对MapReduce执行时间产生负面影响。但是，这种影响可以通过简单地在每个节点上添加更多的处理器来消除。进一步的改进是可能的，如果一个裁缝的地图任务调度程序的代码正在考虑。我们提出了实验和模拟结果，验证这些意见。
英文摘要：
In this paper, we evaluate the efficacy, in a Hadoop setting, of two coding schemes, both possessing an inherent double replication of data. The two coding schemes belong to the class of regenerating and locally regenerating codes respectively, and these two classes are representative of recent advances made in designing codes for the efficient storage of data in a distributed setting. In comparison with triple replication, double replication permits a significant reduction in storage overhead, while delivering good MapReduce performance under moderate work loads. The two coding solutions under evaluation here, add only moderately to the storage overhead of double replication, while simultaneously offering reliability levels similar to that of triple replication. One might expect from the property of inherent data duplication that the performance of these codes in executing a MapReduce job would be comparable to that of double replication. However, a second feature of this class of code comes into play here, namely that under both coding schemes analyzed here, multiple blocks from the same coded stripe are required to be stored on the same node. This concentration of data belonging to a single stripe negatively impacts MapReduce execution times. However, much of this effect can be undone by simply adding a larger number of processors per node. Further improvements are possible if one tailors the Map task scheduler to the codes under consideration. We present both experimental and simulation results that validate these observations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
