以低于8位的算术精度训练大语言模型(LLM)有望大幅提升内存带宽和计算吞吐量,但FP4格式会引入严重的量化误差,若不加以处理会导致优化过程不稳定。我们提出ALTO,这是一个针对GPT-OSS-20B(200亿参数的混合专家(MoE)模型)的开源训练方案,基于MXFP4微缩放格式在MLPerf Small MoE训练基准上实现。该方案结合了Abecassis, Felix等人[arXiv:2509.25149]提出的成熟技术(混合1D/2D块量化、随机哈达玛变换(RHT)和随机舍入(SR)),以及从TetraJet-v2[arXiv:2510.27527]改编的简化权重去振荡方案。在全局批量大小为16的C4验证集上,MXFP4 + RHT + SR +去振荡方案在16128步时达到验证损失3.3350,仅比BF16基线(3.3283)高0.007,缩小了约50%的差距(未使用去振荡的MXFP4的验证损失为3.3418)。以达到MLPerf质量目标(验证损失3.34)所需的步数衡量,去振荡将FP4相对于BF16的收敛开销从+20.0%降至+5.0%。我们还报告了差分梯度估计、异常值裁剪和宏块缩放的负面端到端结果,尽管每种方法都在算子层面显示出信噪比(SNR)提升。所有实验均在AMD MI300硬件上使用模拟的MXFP4内核进行,且未优化挂钟速度。
英文摘要:
Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size 16, MXFP4 + RHT + SR + de-oscillation reaches a validation loss of 3.3350 at 16,128 steps, only 0.007 above the BF16 baseline (3.3283), closing roughly 50% of the gap left by MXFP4 without de-oscillation (3.3418). Measured by steps to reach the MLPerf quality target (validation loss 3.34), de-oscillation cuts the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%. We also report negative end-to-end results from differential gradient estimation, outlier clipping, and macro-block scaling, despite each showing operator-level SNR gains. All experiments use simulated MXFP4 kernels on AMD MI300X hardware and wall-clock speed has not been optimized.
Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs
School of Computer Science, Fudan University(复旦大学计算机科学技术学院);LightSpeed Studios, Tencent(腾讯光速工作室);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学智能信息处理重点实验室);School of Informatics and ELLIS, University of Edinburgh(爱丁堡大学信息学院及欧洲学习与智能系统实验室)
Enhancing Online Recruitment with Category-Aware MoE and LLM-based Data Augmentation
HKUST (GZ)(香港科技大学(广州));HKUST(香港科技大学);Alibaba Group(阿里巴巴集团);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));Zhijiang Lab(浙江实验室);The Hong Kong Polytechnic University(香港理工大学)
When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications
City University of Hong Kong(香港城市大学);Jarvis Research Center(贾维斯研究中心);Tencent YouTu Lab(腾讯优图实验室);Southern University of Science and Technology(南方科技大学);University of Science and Technology of China(中国科学技术大学)
DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devices
School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦女王玛丽大学电子工程与计算机科学学院);Department of Computer Science, Faculty of Environment, Science and Economy, University of Exeter(埃克塞特大学环境、科学与经济学院计算机科学系);School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);Beijing Key Laboratory of Petroleum Data Mining, China University of Petroleum(北京石油数据挖掘重点实验室,中国石油大学)
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院)
Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
Nanjing University of Science and Technology(南京理工大学);C 2 \text{C}^{2} DL, Institute of Automation, Chinese Academy of Sciences(C2 DL,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science, University of Sydney(悉尼大学计算机学院)
CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edge
Harbin Institute of Technology(哈尔滨工业大学);Hong Kong University of Science and Technology(香港科技大学);Guangdong University of Technology(广东工业大学);University of Science and Technology Beijing(北京科技大学)
ProMoE: Fast MoE-based LLM Serving using Proactive Caching
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University(上海交通大学平行与分布式系统研究所);Key Laboratory of System Software (Chinese Academy of Sciences)(中国科学院系统软件重点实验室);Zhejiang University(浙江大学)
A novel YOLO26-MoE optimized by an LLM agent for insulator fault detection considering UAV images
Department of Automation and Systems Engineering, Federal University of Santa Catarina, Florianópolis, Brazil(自动化与系统工程系,圣卡塔琳娜联邦大学,巴西弗洛里安波利斯);Applications Lab, Faculty of Science, University of Salamanca, Plaza de los Caídos s/n, 37008 Salamanca, Spain(应用实验室,科学学院,萨拉曼卡大学,西班牙萨拉曼卡)
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
Guangzhou University(广州大学);Institue of Automation Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance Inc.(字节跳动公司);University of Science and Technology Beijing(北京科技大学)
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院);University of Science and Technology of China(中国科学技术大学)
Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language Models
University of Toronto(多伦多大学);Vector Institute(矢量研究院);Princeton University(普林斯顿大学);McMaster University(麦克马斯特大学);ETH Zürich(苏黎世联邦理工学院);University of Oxford(牛津大学);Dalhousie University(达尔豪斯大学)