arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.02302cs.LGcs.AI
收藏

FP8-Flow-MoE:一种无双重量化误差且免除转换的 FP8 方案

FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error

Zhejiang Lab(浙江实验室)

Fengjuan Wang, Zhiyi Su, Xingzhu Hu, Cheng Wang, Mou Sun

AI总结:

针对大型 MoE 模型低精度训练中的冗余转换与双重量化误差问题,提出 FP8-Flow-MoE 方案,通过量化一致的 FP8 数据流与融合算子将转换操作从 12 个减至 2 个,在 671B 参数模型上实现吞吐量提升 21%、单卡内存降低 16.5 GB。

摘要:

训练大型 Mixture-of-Experts (MoE) 模型由于其极高的计算和内存需求,在计算上仍然令人望而却步。尽管低精度训练有望加速计算并减少内存占用,但现有实现仍依赖于以 BF16 为主的数据流,并伴有频繁的量化-反量化(Q/DQ)转换。这些冗余的转换侵蚀了 FP8 的大部分理论效率。然而,通过保持数据流完全在 FP8 中来天真地移除这些转换会引入双重量化误差:沿不同维度量化的张量会累积不一致的缩放因子,从而降低数值稳定性。我们提出了 FP8-Flow-MoE,这是一种 FP8 训练方案,具有量化一致的以 FP8 为中心的数据流,结合缩放感知转置和融合 FP8 算子,可简化计算并将显式转换操作从 12 个减少到 2 个。在 671B 参数 MoE 模型上的评估表明,与 BF16 和天真 FP8 基线相比,吞吐量提高达 21%,每 GPU 内存使用减少 16.5 GB,同时保持稳定收敛。我们提供与 TransformerEngine 和 Megatron-LM 兼容的即插即用 FP8 方案,即将开源。

英文摘要:

Training large Mixture-of-Experts (MoE) models remains computationally prohibitive due to their extreme compute and memory demands. Although low-precision training promises to accelerate computation and reduce memory footprint, existing implementations still rely on BF16-dominated dataflows with frequent quantize-dequantize (Q/DQ) conversions. These redundant casts erode much of FP8's theoretical efficiency. However, naively removing these casts by keeping dataflows entirely in FP8 introduces double quantization error: tensors quantized along different dimensions accumulate inconsistent scaling factors, degrading numerical stability. We propose FP8-Flow-MoE, an FP8 training recipe featuring a quantization-consistent FP8-centric dataflow with a scaling-aware transpose and fused FP8 operators that streamline computation and eliminate explicit cast operations from 12 to 2. Evaluations on a 671B-parameter MoE model demonstrate up to 21\% higher throughput and 16.5 GB lower memory usage per GPU compared to BF16 and naïve FP8 baselines, while maintaining stable convergence. We provide a plug-and-play FP8 recipe compatible with TransformerEngine and Megatron-LM, which will be open-sourced soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑