SNIP：一种用于子字节大语言模型训练的自适应混合精度框架
SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training
AI总结：
SNIP通过自适应混合精度框架，有效提升大语言模型训练效率，减少FLOPs达80%并保持模型质量。
摘要：
在子字节精度支持下，高效训练大语言模型（LLMs）并保持模型质量面临重大挑战。当前的混合精度训练方法要么对所有GEMM操作应用统一精度，要么依赖启发式方法，这些方法在训练过程中无法推广，导致收敛不充分和不稳定。为了解决这些挑战，本文介绍了SNIP，一种用于LLM预训练的细粒度自适应混合精度训练框架，支持子字节精度。SNIP周期性收集激活、梯度和优化器状态的统计信息，以评估精度损失对模型质量的影响。我们定义了两个关键指标：前向传递中的损失发散，由量化引起的训练损失增加引起，以及反向传递中的权重发散，该指标衡量通过梯度传播的误差对模型更新的影响。这些指标指导一个整数线性规划（ILP）问题，系统地优化每层的精度以最小化总体质量损失，同时满足效率目标。在1B、3B、7B和70B的Llama-like模型上的实验表明，SNIP在不同模型大小和训练阶段中，通过减少FLOPs高达80%的同时保持模型质量，优于现有基线。
英文摘要：
Training large language models (LLMs) efficiently while preserving model quality poses significant challenges, particularly with subbyte precision supported by state-of-the-art GPUs. Current mixed-precision training approaches either apply uniform precision to all GEMM operations or rely on heuristic-based methods that fail to generalize during training, leading to suboptimal convergence and instability. To address these challenges, this paper introduces SNIP, a fine-grained adaptive mixed-precision training framework for LLM pretraining that supports subbyte precision. SNIP periodically collects statistics on activations, gradients, and optimizer states to assess the precision loss impact on model quality. We define two key metrics: loss divergence in the forward pass, caused by quantization-induced increases in training loss, and weight divergence in the backward pass, which measures error propagation through gradients affecting model updates. These metrics guide an Integer Linear Programming (ILP) problem that systematically optimizes layerwise precision to minimize overall quality loss while meeting efficiency targets. Experiments on 1B, 3B, 7B and 70B Llama-like models demonstrate that SNIP consistently outperforms existing baselines, reducing FLOPs by up to 80% while preserving model quality across different model sizes and training phases with minimal computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
