arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1805.08359cs.PF
收藏

DRESS:面向数据密集型计算平台的动态资源预留机制

DRESS: Dynamic RESource-reservation Scheme for Congested Data-intensive Computing Platforms

Ying Mao, Victoria Green, Jiayin Wang, Haoyi Xiong, Zhishan Guo

摘要:

在过去的几年里,我们已经预见到越来越多的企业开始通过大数据分析来推动,比如亚马逊推荐和谷歌广告。在后端,企业由大数据处理平台提供支持,以快速提取信息并做出决策。这些平台运行在计算集群之上,利用调度算法来分配资源。由于有限的资源,例如CPU和内存,以及大量的用户需求,高效的调度器对系统性能至关重要。然而,除了来自客户端的请求和系统的当前状态之外,它对正在运行的作业的执行长度和传入作业的资源需求的了解有限,这使得分配资源成为一项具有挑战性的任务。如果大部分资源都被一个长时间运行的作业占用,其他作业将不得不一直等待,直到它释放它们。本文提出了一种新的调度策略,命名为DRESS,特别是旨在优化之间的分配作业与不同的需求。具体地说,该算法根据作业请求将作业分为两类,为每类作业预留一部分资源,并通过监控待处理请求和估计作业释放模式来动态调整预留比例。实验结果表明,DRESS算法在保持系统整体性能稳定的同时,显著降低了一个类别的完成时间,最高可达76.1%。

英文摘要:

In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑