Pollux：用于产出优化深度学习的自适应集群调度
Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
摘要：
Pollux通过在每个作业级别和整个集群级别自适应地共同优化相互依赖的因素，提高了深度学习（DL）集群中的调度性能。大多数现有的调度器都希望用户为每个作业指定资源数量，这通常会导致资源使用效率低下。一些最近的开发人员为用户选择工作资源，但这样做并不知道如何重新优化DL培训以更好地利用所提供的资源。Pollux同时考虑这两个方面。通过在训练过程中监控每个作业的状态，Pollux对它们的有效吞吐量（我们引入的一种新指标，将系统吞吐量与统计效率相结合）如何通过添加或删除资源而变化进行建模。利用这些信息，Pollux动态（重新）分配资源以提高集群范围的有效吞吐量，同时尊重公平性并不断优化每个DL作业以更好地利用这些资源。在对真实DL作业和跟踪驱动模拟的实验中，Pollux相对于最先进的DL调度器将平均作业完成时间缩短了37-50%，即使为每个作业提供了理想的资源和培训配置。Pollux基于更有意义的有用作业进度度量，促进了竞争资源的DL作业之间的公平性，并揭示了降低云环境中DL成本的新机会。Pollux是作为一个开源项目的一部分在https://github.com/petuum/adaptdl上实现和公开的。
英文摘要：
Pollux improves scheduling performance in deep learning (DL) clusters by adaptively co-optimizing inter-dependent factors both at the per-job level and at the cluster-wide level. Most existing schedulers expect users to specify the number of resources for each job, often leading to inefficient resource use. Some recent schedulers choose job resources for users, but do so without awareness of how DL training can be re-optimized to better utilize the provided resources. Pollux simultaneously considers both aspects. By monitoring the status of each job during training, Pollux models how their goodput (a novel metric we introduce that combines system throughput with statistical efficiency) would change by adding or removing resources. Leveraging these information, Pollux dynamically (re-)assigns resources to improve cluster-wide goodput, while respecting fairness and continually optimizing each DL job to better utilize those resources. In experiments with real DL jobs and with trace-driven simulations, Pollux reduces average job completion times by 37-50% relative to state-of-the-art DL schedulers, even when they are provided with ideal resource and training configurations for every job. Pollux promotes fairness among DL jobs competing for resources based on a more meaningful measure of useful job progress, and reveals a new opportunity for reducing DL cost in cloud environments. Pollux is implemented and publicly available as part of an open-source project at https://github.com/petuum/adaptdl.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
