及时冻结:用于流水线并行的自适应参数冻结机制
TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism
AI总结:
TimelyFreeze通过自适应参数冻结机制优化流水线并行,提升训练吞吐量并保持精度
摘要:
流水线并行使训练能够超越单设备内存的模型成为可能,但实际吞吐量仍受限于流水线气泡。尽管参数冻结可以通过自适应地跳过反向计算来提高训练吞吐量,但现有方法往往过度冻结参数,导致不必要的精度下降。为了解决这个问题,我们提出了TimelyFreeze,它将流水线调度建模为有向无环图,并通过求解线性规划来计算最优冻结比例,以在精度约束下最小化批量执行时间。实验显示,TimelyFreeze在LLaMA-8B上实现了高达40%的训练吞吐量提升,同时保持了可比的精度。总体而言,它使大规模模型训练更快,而不影响收敛性和泛化能力,并在多种流水线并行设置中具有通用性。
英文摘要:
Pipeline parallelism enables training models that exceed single-device memory, but practical throughput remains limited by pipeline bubbles. Although parameter freezing can improve training throughput by adaptively skipping backward computation, existing methods often over-freeze parameters, resulting in unnecessary accuracy degradation. To address this issue, we propose TimelyFreeze, which models the pipeline schedule as a directed acyclic graph and solves a linear program to compute optimal freeze ratios that minimize batch execution time under accuracy constraints. Experiments show that TimelyFreeze achieves up to 40% training throughput improvement on LLaMA-8B with comparable accuracy. Overall, it enables faster large-scale model training without compromising convergence and generalizes across diverse pipeline-parallel settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。