arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.02803cs.LGcs.DC
收藏

Flash Attention 是否稳定?

Is Flash Attention Stable?

FAIR at Meta(Meta FAIR实验室);Harvard University(哈佛大学);Meta

Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, Carole-Jean Wu

AI总结:

本文提出分析数值偏差的框架并应用于 Flash Attention,发现其 BF16 前向传播数值偏差比基线高约一个数量级,但通过 Wasserstein 距离分析表明其对训练权重的影响比低精度训练小 2-5 倍。

摘要:

鉴于当今工作负载的规模和复杂性,训练大规模机器学习模型带来了独特的系统挑战。最近,许多训练最先进生成式 AI 模型的组织报告了训练过程中出现不稳定的情况,通常表现为损失尖峰。数值偏差已成为这种训练不稳定的潜在原因,尽管由于训练运行成本高昂,量化这一点尤其具有挑战性。在这项工作中,我们开发了一种有原则的方法来理解数值偏差的影响,并构建代理指标,以便在下游影响难以量化时将观察结果置于背景中。作为案例研究,我们应用该框架来分析被广泛采用的 Flash Attention 优化。我们发现,在 BF16 下以孤立前向传播进行测量时,与 Baseline Attention 相比,Flash Attention 的数值偏差大约高出一个数量级。然后,我们使用基于 Wasserstein Distance 的数据驱动分析,给出该数值偏差在训练期间对模型权重影响的上界,发现 Flash Attention 中存在的数值偏差比低精度训练的影响小 2-5 倍。

英文摘要:

Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context when downstream effects are difficult to quantify. As a case study, we apply this framework to analyze the widely-adopted Flash Attention optimization. We find that Flash Attention sees roughly an order of magnitude more numeric deviation as compared to Baseline Attention at BF16 when measured during an isolated forward pass. We then use a data-driven analysis based on the Wasserstein Distance to provide upper bounds on how this numeric deviation impacts model weights during training, finding that the numerical deviation present in Flash Attention is 2-5 times less significant than low-precision training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑