arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.10075cs.CL
收藏

RoCoFT:基于行列更新的大型语言模型高效微调

RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

Nokia Bell Labs(诺基亚贝尔实验室);University of Central Florida(中央佛罗里达大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校)

Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Chen Chen, Mojtaba Soltanalian, Niloofar Yousefi

AI总结:

提出RoCoFT,一种仅更新transformers权重矩阵少数行列的大规模语言模型参数高效微调方法,在多种模型上表现出与先进PEFT方法相当或更优的准确率且更高效,并从神经切核理论角度解释了其有效性。

摘要:

我们提出了RoCoFT,一种基于仅更新transformers中权重矩阵的少数行和列的大规模语言模型(LMs)参数高效微调方法。通过对BERT和RoBERTa等中等规模LMs,以及Bloom-7B、Llama2-7B和Llama2-13B等更大规模LMs进行广泛实验,我们表明该方法在提供与最先进PEFT方法相当或更好的准确率的同时,在内存和计算上更加高效。我们还利用神经切核理论的工具研究了该方法有效性的原因。我们实证表明,使用受限的行和列参数集构建的核在数值上接近全参数核,并给出相当的分类性能。我们进行了消融研究,以调查不同算法选择的影响,包括行和列的选择策略以及有效实现该方法的最佳秩。

英文摘要:

We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑