Unicron:规模化自我修复 LLM 训练的经济学优化
Unicron: Economizing Self-Healing LLM Training at Scale
AI总结:
本文提出工作负载管理器 Unicron,通过带内错误检测、动态成本感知计划生成和高效转换策略,最小化集群内多并发任务的故障成本,在 128-GPU 集群上较 SOTA 提升 1.9 倍训练效率,增强大规模 LLM 训练可靠性。
摘要:
在各个领域,训练大规模语言模型日益关键,但频繁的故障阻碍了这一进程,导致显著的时间和经济成本。在基于云的环境中,当前的故障恢复方法未能充分应对出现的多样且复杂的场景,仅狭隘地关注消除单个任务的停机时间,而未考虑对集群的整体成本影响。我们引入了 Unicron,这是一种为大规模语言模型训练中高效自我修复而设计的工作负载管理器。Unicron 通过最小化集群内多个并发任务中与故障相关的成本来优化训练过程。其关键特性包括:用于在不产生额外开销的情况下实时识别错误的带内错误检测、用于最优重配置的动态成本感知计划生成机制,以及用于减少状态转换期间停机时间的高效转换策略。在 128-GPU 分布式集群上部署后,Unicron 展现出比 state-of-the-art 方法高达 1.9 倍的训练效率提升,显著降低了故障恢复成本并增强了大规模语言模型训练的可靠性。
英文摘要:
Training large-scale language models is increasingly critical in various domains, but it is hindered by frequent failures, leading to significant time and economic costs. Current failure recovery methods in cloud-based settings inadequately address the diverse and complex scenarios that arise, focusing narrowly on erasing downtime for individual tasks without considering the overall cost impact on a cluster. We introduce Unicron, a workload manager designed for efficient self-healing in large-scale language model training. Unicron optimizes the training process by minimizing failure-related costs across multiple concurrent tasks within a cluster. Its key features include in-band error detection for real-time error identification without extra overhead, a dynamic cost-aware plan generation mechanism for optimal reconfiguration, and an efficient transition strategy to reduce downtime during state changes. Deployed on a 128-GPU distributed cluster, Unicron demonstrates up to a 1.9x improvement in training efficiency over state-of-the-art methods, significantly reducing failure recovery costs and enhancing the reliability of large-scale language model training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。