大数据处理中的算法比较
Comparisons of Algorithms in Big Data Processing
摘要：
并行计算是Hadoop中MapReduce框架的基础。每个数据块在3台服务器上复制，以提高数据的可用性并降低数据丢失的概率。因此，将Map任务存储在其磁盘上的3台服务器是处理它们的最快服务器，称为本地服务器。与本地服务器在同一机架中的所有服务器称为机架本地服务器，其比本地服务器慢，因为与地图任务相关联的数据块应通过机架顶部交换机获取。所有其他服务器称为远程服务器，是最慢的服务器，因为它们需要从另一个机架中的本地服务器获取数据，因此数据应通过至少2个架顶交换机和一个核心交换机传输。请注意，数据传输路径中的交换机数量取决于数据中心的内部网络结构。先进先出（FIFO）和Hadoop公平竞争（HFS）算法没有考虑数据中心的机架结构，因此已知它们不是重流量延迟最优的，甚至不是吞吐量最优的。考虑到数据中心的机架结构和服务器的异构性，数据中心调度的最新进展导致了最先进的Balanced-PANDAS算法，其性能优于经典的MaxWeight算法。在Balanced-PANDAS和MaxWeight算法中，假设本地、机架本地和远程服务器的处理速率是已知的。然而，除了处理速率的估计误差之外，随着业务量随时间的变化，认为处理速率是已知的是不现实的。在这项工作中，我们研究了平衡PANDAS和MaxWeight算法的鲁棒性不准确的估计处理速率。我们观察到，Balanced-PANDAS在处理速率的准确性上不像MaxWeight那样敏感，这使得它更适合在数据中心使用。
英文摘要：
Parallel computing is the fundamental base for MapReduce framework in Hadoop. Each data chunk is replicated over 3 servers for increasing availability of data and decreasing probability of data loss. Hence, the 3 servers that have Map task stored on their disk are fastest servers to process them, which are called local servers. All servers in the same rack as local servers are called rack-local servers that are slower than local servers since data chunk associated with Map task should be fetched through top of the rack switch. All other servers are called remote servers that are slowest servers since they need to fetch data from a local server in another rack, so data should be transmitted through at least 2 top of rack switches and a core switch. Note that number of switches in path of data transfer depends on internal network structure of data centers. The First-In-First-Out (FIFO) and Hadoop Fair Scheduler (HFS) algorithms do not take rack structure of data centers into account, so they are known to not be heavy-traffic delay optimal or even throughput optimal. The recent advances on scheduling for data centers considering rack structure of them and heterogeneity of servers resulted in state-of-the-art Balanced-PANDAS algorithm that outperforms classic MaxWeight algorithm. In both Balanced-PANDAS and MaxWeight algorithms, processing rate of local, rack-local, and remote servers are assumed to be known. However, with the change of traffic over time in addition to estimation errors of processing rates, it is not realistic to consider processing rates to be known. In this work, we study robustness of Balanced-PANDAS and MaxWeight algorithms in terms of inaccurate estimations of processing rates. We observe that Balanced-PANDAS is not as sensitive as MaxWeight on the accuracy of processing rates, making it more appealing to use in data centers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
