解构LLM NVFP4预训练中的异常动态
Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
AI总结：
本研究通过分析NVFP4预训练中的异常动态，提出HCP机制和CHON训练配方，有效缩小了与BF16的损失差距。
摘要：
使用4位算术训练大型语言模型可以提高吞吐量和内存效率。然而，FP4有限的动态范围会增加对异常值的敏感性。虽然NVFP4通过分层微缩放来减轻量化误差，但与BF16相比仍存在持续的损失差距。本研究对NVFP4预训练过程中架构中的异常动态进行了纵向分析，重点关注它们的定位位置、产生原因以及时间上的演变过程。我们发现，与Softmax注意力（SA）相比，线性注意力（LA）减少了每个张量的重尾分布，但在块量化下仍表现出持续的块级尖峰。我们的分析将异常归因于特定的架构组件：SA中的Softmax、LA中的门控以及FFN中的SwiGLU，其中
英文摘要：
Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining, focusing on where they localize, why they occur, and how they evolve temporally. We find that, compared with Softmax Attention (SA), Linear Attention (LA) reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization. Our analysis attributes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with "post-QK" operations exhibiting higher sensitivity to quantization. Notably, outliers evolve from transient spikes early in training to a small set of persistent hot channels (i.e., channels with persistently large magnitudes) in later stages. Based on these findings, we introduce Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels and reinjects residuals using hardware-efficient kernels. We then develop CHON, an NVFP4 training recipe integrating HCP with post-QK operation protection. On GLA-1.3B model trained for 60B tokens, CHON reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
