arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.16626cs.LGcs.AIcs.AR
收藏

用于深度学习的Ascend HiFloat8格式

Ascend HiFloat8 Format for Deep Learning

HiSilicon, Huawei(华为海思);Huawei(华为);Central Hardware, Huawei(华为中央硬件部);Computing Product Line, Huawei(华为计算产品线);Central Media, Huawei(华为中央媒体部)
Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, … 展开作者

Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, Mikhaylov Dmitry, Korviakov Vladimir, Bobrin Maxim, Yuhao Hu, Guanfu Chen, Zeyi Huang

AI总结:

提出8位浮点格式HiFloat8(HiF8),通过锥形精度平衡精度与动态范围,支持AI训练前向与反向传播,并在多种神经网络上验证其有效性。

摘要:

本初步白皮书提出了一种用于深度学习的8位浮点数据格式HiFloat8(缩写为HiF8)。HiF8具有锥形精度。对于规格化值编码,它提供3位尾数的7个指数值、2位尾数的8个指数值和1位尾数的16个指数值。对于非规格化值编码,它将动态范围扩展了额外的7个2的幂,从31个binades扩展到38个binades(注意FP16覆盖40个binades)。同时,HiF8编码所有特殊值,只是正零和负零仅由一个位模式表示。由于在精度和动态范围之间取得了更好的平衡,HiF8可同时用于AI训练的前向和反向传播。本文将描述HiF8的定义和舍入方法,以及初步的训练和推理解决方案。为展示HiF8的有效性,还将呈现包括传统神经网络和大语言模型(LLMs)在内的各种神经网络的大规模仿真结果。

英文摘要:

This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponent values with 3-bit mantissa, 8 exponent values with 2-bit mantissa, and 16 exponent values with 1-bit mantissa. For denormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑