Training large Mixture-of-Experts (MoE) models remains computationally prohibitive due to their extreme compute and memory demands. Although low-precision training promises to accelerate computation and reduce memory footprint, existing implementations still rely on BF16-dominated dataflows with frequent quantize-dequantize (Q/DQ) conversions. These redundant casts erode much of FP8's theoretical efficiency. However, naively removing these casts by keeping dataflows entirely in FP8 introduces double quantization error: tensors quantized along different dimensions accumulate inconsistent scaling factors, degrading numerical stability.
We propose FP8-Flow-MoE, an FP8 training recipe featuring a quantization-consistent FP8-centric dataflow with a scaling-aware transpose and fused FP8 operators that streamline computation and eliminate explicit cast operations from 12 to 2. Evaluations on a 671B-parameter MoE model demonstrate up to 21\% higher throughput and 16.5 GB lower memory usage per GPU compared to BF16 and naïve FP8 baselines, while maintaining stable convergence. We provide a plug-and-play FP8 recipe compatible with TransformerEngine and Megatron-LM, which will be open-sourced soon.
$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);MIT(麻省理工学院);Harvard University(哈佛大学);The Chinese University of Hong Kong(香港中文大学)
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE(Mohamed bin Zayed人工智能大学(MBZUAI));Michigan State University (MSU), USA(密歇根州立大学(MSU));Amazon Science, UK(亚马逊科学)
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
University of Utah(犹他大学);University of Waterloo(滑铁卢大学);The University of Queensland(昆士兰大学);Carnegie Mellon University(卡内基梅隆大学);University of Virginia(弗吉尼亚大学)
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA(计算机科学与软件工程系,阿伯拉罕大学,阿伯丁,阿拉巴马州,美国);Department of Information Management, National Central University, Taoyuan, Taiwan(信息管理系,国立中央大学,桃园,台湾)
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
PleIAs;Sorbonne Center for Artificial Intelligence(索邦人工智能中心);Sciences Po Médialab(巴黎政治学院媒体实验室);EPFL(瑞士洛桑联邦理工学院);CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt(维尔茨堡-施韦因富特应用技术大学CAIRO);Paris Dauphine-PSL(巴黎多芬纳-PSL大学);Lattice, ENS-PSL(巴黎高等师范学院-巴黎文理研究大学Lattice实验室);TU Munich, Munich Center for Machine Learning(慕尼黑工业大学慕尼黑机器学习中心);KU Leuven(鲁汶大学)
Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
Shanghai Jiao Tong University(上海交通大学);SenseTime Research(商汤科技研究院);Central South University(中南大学);The Chinese University of Hong Kong(香港中文大学);Beihang University(北京航空航天大学)
SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
Leap Lab, Tsinghua University(清华大学 Leap 实验室);Qwen Large Model Application Team, Alibaba(阿里巴巴 Qwen 大模型应用团队);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息学研究院)