arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.08708cs.CL
收藏

语言模型微调的词表级内存效率优化

Vocabulary-level Memory Efficiency for Language Model Fine-tuning

University of Sheffield(谢菲尔德大学)

Miles Williams, Nikolaos Aletras

AI总结:

针对语言模型微调内存占用大的问题,本文发现词表中大量词汇未被使用,据此提出减少嵌入矩阵内存占用的方法,在不影响下游任务性能的前提下显著降低内存使用。

摘要:

语言模型(LM)微调过程中庞大的内存占用对研究者和从业者都构成了挑战。语言模型使用嵌入矩阵来表示庞大的词表,这占据了模型参数的很大比例。以往针对内存高效微调的研究主要集中于最小化可训练参数的数量,但减少嵌入矩阵的内存占用尚未被探索。我们首先证明,在微调期间有相当大比例的词汇未被使用。随后,我们提出了一种简单而有效的方法,利用这一发现来最小化内存使用。我们表明,我们的方法在广泛的模型和任务中显著减少了内存占用。值得注意的是,我们的方法不会影响下游任务性能,同时允许更高效地利用计算资源。

英文摘要:

The extensive memory footprint of language model (LM) fine-tuning poses a challenge for both researchers and practitioners. LMs use an embedding matrix to represent extensive vocabularies, forming a substantial proportion of the model parameters. While previous work towards memory-efficient fine-tuning has focused on minimizing the number of trainable parameters, reducing the memory footprint of the embedding matrix has yet to be explored. We first demonstrate that a significant proportion of the vocabulary remains unused during fine-tuning. We then propose a simple yet effective approach that leverages this finding to minimize memory usage. We show that our approach provides substantial reductions in memory usage across a wide range of models and tasks. Notably, our approach does not impact downstream task performance, while allowing more efficient use of computational resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑