arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.08181cs.CL
收藏

方向就是你所需要的:改进大语言模型中的词嵌入压缩

Direction is what you need: Improving Word Embedding Compression in Large Language Models

Jagiellonian University(雅盖隆大学);EPFL(洛桑联邦理工学院)

Klaudia Bałazy, Mohammadreza Banaei, Rémi Lebret, Jacek Tabor, Karl Aberer

摘要:

在自然语言处理(NLP)中采用基于transformer的模型,使用大量参数取得了巨大成功。然而,由于边缘设备中的部署限制,人们对压缩这些模型以改善其推理时间和内存占用的兴趣越来越大。本文提出了一种新的损失目标,通过利用AutoEncoder架构来压缩基于Transformer的模型中的令牌嵌入。更具体地说,我们强调压缩嵌入的方向相对于原始未压缩嵌入的重要性。所提出的方法是任务不可知的,不需要进一步的语言建模预训练。在初始语言模型复杂度方面,我们的方法明显优于常用的基于SVD的矩阵分解方法。此外,我们在SQuAD v1.1数据集和GLUE基准测试的几个下游任务上评估了我们提出的方法,在大多数情况下,我们的性能也优于基线。代码是公开的。

英文摘要:

The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of these models to improve their inference time and memory footprint. This paper presents a novel loss objective to compress token embeddings in the Transformer-based models by leveraging an AutoEncoder architecture. More specifically, we emphasize the importance of the direction of compressed embeddings with respect to original uncompressed embeddings. The proposed method is task-agnostic and does not require further language modeling pre-training. Our method significantly outperforms the commonly used SVD-based matrix-factorization approach in terms of initial language model Perplexity. Moreover, we evaluate our proposed approach over SQuAD v1.1 dataset and several downstream tasks from the GLUE benchmark, where we also outperform the baseline in most scenarios. Our code is public.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑