在预训练期间移除异常值是免费的午餐吗?
Is It a Free Lunch for Removing Outliers during Pretraining?
AI总结:
针对无异常值预训练方法在全精度下性能下降的问题,通过引入对序列长度不变的归一化改进该方法,不仅弥合了预训练与微调的差距,还成功支持了因果语言模型的预训练。
摘要:
随着大语言模型规模的不断增长,量化的作用变得日益重要。然而,权重或激活中存在的异常值会显著影响量化模型的性能。最近,\citet{qtransformer}引入了一种新颖的softmax函数,旨在以无异常值的方式预训练模型,从而增强其对量化的适用性。有趣的是,我们观察到这种方法会导致全精度下的性能下降。基于这一洞察,我们通过确保其归一化对序列长度保持不变来改进该方法,这是弥合预训练和微调之间差距的关键因素。此外,这种改进的方法也促进了因果语言模型的成功预训练。
英文摘要:
With the growing size of large language models, the role of quantization becomes increasingly significant. However, outliers present in weights or activations notably influence the performance of quantized models. Recently, \citet{qtransformer} introduced a novel softmax function aimed at pretraining models in an outlier-free manner, thereby enhancing their suitability for quantization. Interestingly, we observed that such an approach leads to performance degradation in full precision. Building on this insight, we enhance the method by ensuring its normalization is invariant to sequence length, a crucial factor for bridging the gap between pretraining and fine-tuning. Moreover, this improved method also facilitates successful pretraining of causal language models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。