Pretraining transformers are generally time-consuming. Fully quantized training (FQT) is a promising approach to speed up pretraining. However, most FQT methods adopt a quantize-compute-dequantize procedure, which often leads to suboptimal speedup and significant performance degradation when used in transformers due to the high memory access overheads and low-precision computations. In this work, we propose Jetfire, an efficient and accurate INT8 training method specific to transformers. Our method features an INT8 data flow to optimize memory access and a per-block quantization method to maintain the accuracy of pretrained transformers. Extensive experiments demonstrate that our INT8 FQT method achieves comparable accuracy to the FP16 training baseline and outperforms the existing INT8 training works for transformers. Moreover, for a standard transformer block, our method offers an end-to-end training speedup of 1.42x and a 1.49x memory reduction compared to the FP16 baseline. Our code is open sourced at https://github.com/thu-ml/Jetfire-INT8Training.
Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
University of California San Diego（加利福尼亚大学圣地亚哥分校）
现代神经网络的深层在输入通过网络传播时提取相当丰富的特征集。本文旨在以最小的精度损失收获这些丰富的中间表示进行量化，同时显着降低DNN的内存占用和计算强度。本文通过师生范式利用知识蒸馏（Hinton等人，2015）在一种新颖的设置中，该设置利用DNN的特征提取能力进行更高精度的量化。因此，我们的算法在逻辑上将预训练的全精度DNN划分为多个部分，每个部分都暴露了中间特征，以便在量化域中独立地训练一组学生。事实上，这种分而治之的策略使得每个学生部分的训练可以孤立进行，而所有这些独立训练的部分随后被缝合在一起以形成等效的完全量化的网络。我们的算法是一种面向知识蒸馏的分段方法，并且在整个网络上进行一次知识蒸馏之前，不将中间表示视为预训练的提示（Romero等人，2015年）。在各种DNN（AlexNet、LeNet、MobileNet、ResNet-18、ResNet-20、SVHN和VGG-11）上的实验表明，这种称为DCQ（Divide and Conquer Quantization）的方法平均而言提高了最先进的量化训练技术DoReFa-Net（Zhou et al.，2016年），分别为21.6%和9.3%的二进制和三进制量化。此外，我们表明，将DCQ纳入现有的量化训练方法，导致提高精度相比，以前报道的多个国家的最先进的量化训练方法。
