Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context when downstream effects are difficult to quantify. As a case study, we apply this framework to analyze the widely-adopted Flash Attention optimization. We find that Flash Attention sees roughly an order of magnitude more numeric deviation as compared to Baseline Attention at BF16 when measured during an isolated forward pass. We then use a data-driven analysis based on the Wasserstein Distance to provide upper bounds on how this numeric deviation impacts model weights during training, finding that the numerical deviation present in Flash Attention is 2-5 times less significant than low-precision training.
FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs
Tencent(腾讯);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);College of Artificial Intelligence, Shenzhen Polytechnic University(深圳职业技术学院人工智能学院)
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration
Dept. of Comp. Sci. & Tech.(计算机科学与技术系);Institute for AI(人工智能研究院);BNRist Center(BNRist中心);Tsinghua-Bosch Joint ML Center(清华大学-博世联合机器学习中心);THBI Lab(THBI实验室);Tsinghua University(清华大学)
Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
College of Future Information Technology, Fudan University(未来信息科技学院,复旦大学);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhangjiang Laboratory(张江实验室)
Efficiently Dispatching Flash Attention For Partially Filled Attention Masks
University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);ELLIS Institute Tübingen(ELLIS蒂宾根研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);State Key laboratory of Space Network and Communications(空间网络与通信国家重点实验室)
SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
Dept. of Comp. Sci. and Tech.(计算机科学与技术系);Institute for AI(人工智能研究院);BNRist Center(BNRist中心);THBI Lab(THBI实验室);Tsinghua-Bosch Joint ML Center(清华大学-博世联合机器学习中心);Tsinghua University(清华大学)