arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.32229cs.AI
收藏

以MXFP4格式训练MoE大语言模型:我们如何缩小与BF16的差距

Training an MoE LLM in MXFP4: How We Closed the Gap to BF16

Advanced Micro Devices, Inc.(超威半导体公司)

Han Wang, Zhitao Wang, Natalia Frumkin, Xuanwu Yin, Alireza Khodamoradi, Dong Li, Emad Barsoum

AI总结:

本研究提出ALTO训练方案,结合多种量化技术与去振荡方案,在MXFP4格式下训练200亿参数MoE LLM,大幅缩小了与BF16的性能差距,降低了FP4的收敛开销。

摘要:

以低于8位的算术精度训练大语言模型(LLM)有望大幅提升内存带宽和计算吞吐量,但FP4格式会引入严重的量化误差,若不加以处理会导致优化过程不稳定。我们提出ALTO,这是一个针对GPT-OSS-20B(200亿参数的混合专家(MoE)模型)的开源训练方案,基于MXFP4微缩放格式在MLPerf Small MoE训练基准上实现。该方案结合了Abecassis, Felix等人[arXiv:2509.25149]提出的成熟技术(混合1D/2D块量化、随机哈达玛变换(RHT)和随机舍入(SR)),以及从TetraJet-v2[arXiv:2510.27527]改编的简化权重去振荡方案。在全局批量大小为16的C4验证集上,MXFP4 + RHT + SR +去振荡方案在16128步时达到验证损失3.3350,仅比BF16基线(3.3283)高0.007,缩小了约50%的差距(未使用去振荡的MXFP4的验证损失为3.3418)。以达到MLPerf质量目标(验证损失3.34)所需的步数衡量,去振荡将FP4相对于BF16的收敛开销从+20.0%降至+5.0%。我们还报告了差分梯度估计、异常值裁剪和宏块缩放的负面端到端结果,尽管每种方法都在算子层面显示出信噪比(SNR)提升。所有实验均在AMD MI300硬件上使用模拟的MXFP4内核进行,且未优化挂钟速度。

英文摘要:

Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size 16, MXFP4 + RHT + SR + de-oscillation reaches a validation loss of 3.3350 at 16,128 steps, only 0.007 above the BF16 baseline (3.3283), closing roughly 50% of the gap left by MXFP4 without de-oscillation (3.3418). Measured by steps to reach the MLPerf quality target (validation loss 3.34), de-oscillation cuts the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%. We also report negative end-to-end results from differential gradient estimation, outlier clipping, and macro-block scaling, despite each showing operator-level SNR gains. All experiments use simulated MXFP4 kernels on AMD MI300X hardware and wall-clock speed has not been optimized.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

相关深度报道

↑