LLMTailor: 一种用于大型语言模型高效检查点存储的分层定制工具
LLMTailor: A Layer-wise Tailoring Tool for Efficient Checkpointing of Large Language Models
AI总结:
LLMTailor通过分层定制工具实现大型语言模型高效检查点存储,减少存储开销和检查点时间,同时保持模型质量。
摘要:
检查点对于训练大型语言模型(LLMs)的容错性至关重要。然而,现有方法无论其I/O策略如何,都会周期性地存储整个模型和优化器状态,导致显著的存储开销和资源竞争。最近的研究表明,LLM各层的更新高度非均匀。在训练步骤中,某些层可能经历更显著的变化,而其他层则相对稳定或甚至不变。这表明,仅选择性地检查点具有显著更新的层可以减少开销而不影响训练。实现此类选择性策略需要对权重和优化器状态进行细粒度控制,而目前没有任何工具提供。为了解决这一差距,我们提出了LLMTailor,一种检查点合并框架,用于过滤和组装不同检查点中的层以形成复合检查点。我们的评估表明,LLMTailor可以与不同的选择性检查点策略配合使用,并有效减少检查点大小(例如,对于Llama3.1-8B,检查点大小减少4.3倍)和检查点时间(例如,对于Qwen2.5-7B,检查点时间加快2.8倍),同时保持模型质量。
英文摘要:
Checkpointing is essential for fault tolerance in training large language models (LLMs). However, existing methods, regardless of their I/O strategies, periodically store the entire model and optimizer states, incurring substantial storage overhead and resource contention. Recent studies reveal that updates across LLM layers are highly non-uniform. Across training steps, some layers may undergo more significant changes, while others remain relatively stable or even unchanged. This suggests that selectively checkpointing only layers with significant updates could reduce overhead without harming training. Implementing such selective strategies requires fine-grained control over both weights and optimizer states, which no current tool provides. To address this gap, we propose \texttt{LLMTailor}, a checkpoint-merging framework that filters and assembles layers from different checkpoints to form a composite checkpoint. Our evaluation indicates that LLMTailor can work with different selective checkpointing strategies and effectively reduce checkpoint size (e.g., 4.3 times smaller for Llama3.1-8B) and checkpoint time (e.g., 2.8 times faster for Qwen2.5-7B) while maintaining model quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。