SAF：Hadoop集群的模拟退火公平调度算法Yarn
SAF: Simulated Annealing Fair Scheduling for Hadoop Yarn Clusters
摘要：
Apache引入了YARN作为下一代Hadoop框架，提供资源管理和中央平台，以跨Hadoop集群提供一致的数据治理工具。Hadoop YARN支持MapReduce等多个框架来处理不同类型的数据，并使用不同的调度策略，如FIFO，容量和公平调度器。DRF是使用短期的、不考虑历史信息的、收敛到多类型资源分配的公平性的最佳选择。然而，DRF性能仍然不令人满意，由于公平性和性能之间的权衡资源利用。为了解决这个问题，我们提出了模拟退火公平调度，SAF，一个长期的公平计划在资源分配具有公平性和优异的性能，在资源利用率和MakeSpan。我们引入了一个新的参数熵，这是一种方法来表示整个集群的资源分配的公平性的混乱。我们在Hadoop Yarn集群中实现了SAF作为一个可插拔的调度器，并在Yarn集群负载模拟器（SLS）和CloudSim Plus模拟框架中使用标准的MapReduce基准对其进行了评估。最后，两个模拟工具的结果都证明了我们的主张。与DRF相比，SAF显著提高了YARN集群的资源利用率，并将MakeSpan降低到适当的水平。
英文摘要：
Apache introduced YARN as the next generation of the Hadoop framework, providing resource management and a central platform to deliver consistent data governance tools across Hadoop clusters. Hadoop YARN supports multiple frameworks like MapReduce to process different types of data and works with different scheduling policies such as FIFO, Capacity, and Fair schedulers. DRF is the best option that uses short-term, without considering history information, convergence to fairness for multi-type resource allocation. However, DRF performance is still not satisfying due to trade-offs between fairness and performance regarding resource utilization. To address this problem, we propose Simulated Annealing Fair scheduling, SAF, a long-term fair scheme in resource allocation to have fairness and excellent performance in terms of resource utilization and MakeSpan. We introduce a new parameter as entropy, which is an approach to indicates the disorder in the fairness of allocated resources of the whole cluster. We implemented SAF as a pluggable scheduler in Hadoop Yarn Cluster and evaluated it with standard MapReduce benchmarks in Yarn Scheduler Load Simulator (SLS) and CloudSim Plus simulation framework. Finally, the results of both simulation tools are evidence to prove our claim. Compared to DRF, SAF increases resource utilization of YARN clusters significantly and decreases MakeSpan to an appropriate level.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
