基于结构化子空间随机性的随机舍入高效量化训练
Efficient Quantization Training with Stochastic Rounding via Structured Subspace Randomness
AI总结:
针对随机舍入量化训练中阈值共享导致的收敛退化与速度瓶颈,提出S3R及S3R-EC误差修正方法,在TPU v5p上实现INT8训练的显著加速且收敛性匹配独立随机舍入。
摘要:
低精度训练通常依赖随机舍入(SR)以保持量化梯度无偏,并防止小更新被舍入为零。标准SR为每个梯度元素绘制独立随机阈值,但加速器在向量单元上生成随机数的吞吐量远落后于矩阵单元,为每个元素绘制一个阈值会使向量单元饱和,导致带SR的INT8训练比BF16训练更慢。跨张量元素共享随机阈值可消除此成本,但会迫使这些元素沿同一方向舍入,并使舍入误差产生共同偏移。研究发现,该偏移主要通过激活梯度损害训练,激活梯度会将误差反向传播到更早的层,而更早的层无法用自身局部舍入误差抵消传入的偏移,因此必须在偏移产生的层处将其移除。本文提出结构化子空间随机舍入(S3R),该方法对每个上游梯度量化一次,用于激活和权重梯度矩阵乘法,并在隐藏维度上为每个token共享单个阈值;其误差修正版本S3R-EC从激活和权重梯度中减去结构化平均舍入误差,同时保持优化器不变。在大语言模型(LLM)预训练实验中,未修正的阈值共享会降低收敛性,而S3R-EC可消除此退化,且与独立SR的验证损失相当;此外,在TPU v5p上,S3R-EC使密集LLM的全INT8训练步骤比BF16加速1.24至1.32倍,比独立SR加速1.11倍。
英文摘要:
Low-precision training often relies on stochastic rounding (SR) to keep quantized gradients unbiased and prevent small updates from rounding to zero. Standard SR draws an independent random threshold for every gradient element. However, accelerators generate random numbers on vector units whose throughput lags far behind their matrix units. Drawing one threshold per element saturates the vector unit and can make INT8 training with SR slower than BF16 training. Sharing random thresholds across tensor elements removes this cost but forces those elements to round in the same direction and shifts their rounding errors by a common offset. We find that this offset degrades training primarily through the activation gradient, which propagates the error backward into earlier layers. Because earlier layers cannot cancel an incoming offset using their own local rounding errors, the offset must be removed at the layer where it originates. We propose Structured Subspace Stochastic Rounding (S3R), which quantizes each upstream gradient once for both the activation and weight gradient matrix multiplications and shares a single threshold per token across the hidden dimension. Its error correction, S3R-EC, subtracts the structured mean rounding error from the activation and weight gradients while leaving the optimizer unchanged. In LLM pre-training experiments, uncorrected threshold sharing degrades convergence, whereas S3R-EC eliminates this degradation and matches the validation loss of independent SR. Furthermore, S3R-EC accelerates full INT8 training steps of dense LLMs on TPU v5p by 1.24x to 1.32x over BF16 and by 1.11x over independent SR.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。