arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.05433cs.LG
收藏

用于深度学习的FP8解决方案

FP8 Formats for Deep Learning

NVIDIA(英伟达);Arm(安谋);Intel(英特尔)
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu… 展开作者

Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu

摘要:

FP 8是加速深度学习训练推理的自然进展,超越了现代处理器中常见的16位格式。在本文中,我们提出了一个8位浮点(FP 8)二进制交换格式,由两个编码-E4 M3(4位指数和3位尾数)和E5 M2(5位指数和2位尾数)。虽然E5 M2遵循IEEE 754约定来表示特殊值,但E4 M3的动态范围通过不表示无穷大并且对于NaN仅具有一个尾数位模式来扩展。我们证明了FP 8格式在各种图像和语言任务上的有效性,有效地匹配了16位训练会话所实现的结果质量。我们的研究涵盖了主要的现代神经网络架构-CNN、RNN和基于Transformer的模型,所有超参数都与16位基线训练会话保持不变。我们的训练实验包括多达175 B参数的大型语言模型。我们还研究了使用16位格式训练的语言模型的FP 8后训练量化,这些格式抵抗固定点int 8量化。

英文摘要:

FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings - E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa). While E5M2 follows IEEE 754 conventions for representatio of special values, E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions. Our study covers the main modern neural network architectures - CNNs, RNNs, and Transformer-based models, leaving all the hyperparameters unchanged from the 16-bit baseline training sessions. Our training experiments include large, up to 175B parameter, language models. We also examine FP8 post-training-quantization of language models trained using 16-bit formats that resisted fixed point int8 quantization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑