Classical list scheduling is a very popular and efficient technique for scheduling jobs in parallel and distributed platforms. It is inherently centralized. However, with the increasing number of processors, the cost for managing a single centralized list becomes too prohibitive. A suitable approach to reduce the contention is to distribute the list among the computational units: each processor has only a local view of the work to execute. Thus, the scheduler is no longer greedy and standard performance guarantees are lost.
The objective of this work is to study the extra cost that must be paid when the list is distributed among the computational units. We first present a general methodology for computing the expected makespan based on the analysis of an adequate potential function which represents the load unbalance between the local lists. We obtain an equation on the evolution of the potential by computing its expected decrease in one step of the schedule. Our main theorem shows how to solve such equations to bound the makespan. Then, we apply this method to several scheduling problems, namely, for unit independent tasks, for weighted independent tasks and for tasks with precendence constraints. More precisely, we prove that the time for scheduling a global workload W composed of independent unit tasks on m processors is equal to W/m plus an additional term proportional to log_2 W. We provide a lower bound which shows that this is optimal up to a constant. This result is extended to the case of weighted independent tasks. In the last setting, precedence task graphs, our analysis leads to an improvement on the bound of Arora et al. We finally provide some experiments using a simulator. The distribution of the makespan is shown to fit existing probability laws. The additive term is shown by simulation to be around 3 \log_2 W confirming the tightness of our analysis.
SWARM+: Scalable and Resilient Multi-Agent Consensus for Decentralized Data-Aware Workload Management
Renaissance Computing Institute, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校文艺复兴计算研究所）；Argonne National Laboratory（阿贡国家实验室）；Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）
GADGET: Online Resource Optimization for Scheduling Ring-All-Reduce Learning Jobs
在分布式深度学习（Distributed Deep Learning，简称EML）的推动下，近年来，人们对资源密集型分布式/并行计算的需求迅速增长，以处理EML计算作业。为了解决分布式计算中的网络通信瓶颈和负载均衡问题，越来越多地采用所谓的"环全减“分散式架构来消除对专用参数服务器的需要。然而，到目前为止，仍然缺乏理论上的理解，如何设计资源优化算法，有效地调度计算集群中的环所有减少的并行作业。这促使我们提出了一系列新的资源调度设计，以填补这一空白的环所有减少的可重构作业。我们在本文中的贡献是三方面的：i）我们提出了一个新的资源调度分析模型，用于环全归约深度学习，它涵盖了性能优化中的广泛目标（例如，过度训练避免，能源效率，公平性）; ii）基于所提出的性能分析模型，我们开发了一种高效的资源调度算法，称为GADGET（贪婪环-全部-减少分布式图嵌入技术），它享有可证明的强性能保证; iii）我们进行了广泛的跟踪驱动的实验，以证明GADGET方法的有效性及其优于现有技术的优势。
