重新审视大规模机器学习研究集群中的可靠性
Revisiting Reliability in Large-Scale Machine Learning Research Clusters
AI总结:
本文通过分析两个大规模ML集群中400万个作业和1.5亿GPU小时的数据,建立故障模型并提出有效训练时间比率指标,为提升AI超算集群可靠性提供优化方向。
摘要:
可靠性是运营大规模机器学习基础设施的一项根本性挑战,尤其是随着机器学习模型和训练集群规模的持续增长。尽管对基础设施故障已有数十年的研究,但不同规模下作业故障的影响仍不明确。本文呈现了对两个大型多租户机器学习集群的管理视角,提供了定量分析、运营经验,以及我们在理解和应对大规模可靠性问题方面的自身观点。我们的分析表明,虽然大型作业最容易受到故障影响,但小型作业在集群中占作业总数的大多数,应当被纳入优化目标。我们识别了关键的工作负载特性,在不同集群之间进行了比较,并展示了在大规模条件下推动机器学习训练边界所必需的基本可靠性要求。我们在此引入一套故障分类体系和关键可靠性指标,分析了来自两个先进机器学习环境、历时11个月的数据,涵盖400万个作业和超过1.5亿个A100 GPU小时。基于我们的数据,我们拟合了一个故障模型,用以预测不同GPU规模下的平均故障间隔时间。我们进一步提出了一种方法来估计相关指标——有效训练时间比率,将其作为作业参数的函数,并利用该模型评估潜在软件缓解措施在大规模条件下的有效性。我们的工作为提高AI超级计算机集群的可靠性提供了有价值的见解和未来研究方向,强调了构建灵活、与工作负载无关且具备可靠性感知能力的基础设施、系统软件和算法的必要性。
英文摘要:
Reliability is a fundamental challenge in operating large-scale machine learning (ML) infrastructures, particularly as the scale of ML models and training clusters continues to grow. Despite decades of research on infrastructure failures, the impact of job failures across different scales remains unclear. This paper presents a view of managing two large, multi-tenant ML clusters, providing quantitative analysis, operational experience, and our own perspective in understanding and addressing reliability concerns at scale. Our analysis reveals that while large jobs are most vulnerable to failures, smaller jobs make up the majority of jobs in the clusters and should be incorporated into optimization objectives. We identify key workload properties, compare them across clusters, and demonstrate essential reliability requirements for pushing the boundaries of ML training at scale. We hereby introduce a taxonomy of failures and key reliability metrics, analyze 11 months of data from two state-of-the-art ML environments with 4 million jobs and over 150 million A100 GPU hours. Building on our data, we fit a failure model to project Mean Time to Failure for various GPU scales. We further propose a method to estimate a related metric, Effective Training Time Ratio, as a function of job parameters, and we use this model to gauge the efficacy of potential software mitigations at scale. Our work provides valuable insights and future research directions for improving the reliability of AI supercomputer clusters, emphasizing the need for flexible, workload-agnostic, and reliability-aware infrastructure, system software, and algorithms.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。