基于离散傅里叶变换的参数高效微调
Parameter-Efficient Fine-Tuning with Discrete Fourier Transform
AI总结:
本文提出FourierFT方法,通过仅学习权重变化矩阵的少量频谱系数并利用离散傅里叶逆变换恢复权重更新,以远少于LoRA的参数实现相当或更优的微调性能。
摘要:
低秩适应(LoRA)最近在基础模型微调中引起了广泛关注。它通过引入低秩矩阵$A$和$B$来表示权重变化,即$ΔW=BA$,从而有效减少了可训练参数的数量。尽管LoRA取得了进展,但在处理大量定制化适应或更大的基础模型时,它面临存储挑战。在这项工作中,我们旨在利用傅里叶变换强大的表达能力来进一步压缩可训练参数。具体来说,我们提出了FourierFT,它将$ΔW$视为空间域中的矩阵,并且只学习其一小部分频谱系数。利用训练好的频谱系数,我们执行离散傅里叶逆变换来恢复$ΔW$。在实验上,我们的FourierFT方法在包括自然语言理解、自然语言生成、指令微调和图像分类在内的各种任务上,以比LoRA更少的参数取得了相当或更好的性能。例如,在LLaMA2-7B模型上进行指令微调时,FourierFT仅使用0.064M可训练参数就超越了LoRA,而LoRA的可训练参数为33.5M。我们的代码发布在\url{https://github.com/Chaos96/fourierft}。
英文摘要:
Low-rank adaptation~(LoRA) has recently gained much interest in fine-tuning foundation models. It effectively reduces the number of trainable parameters by incorporating low-rank matrices $A$ and $B$ to represent the weight change, i.e., $ΔW=BA$. Despite LoRA's progress, it faces storage challenges when handling extensive customization adaptations or larger base models. In this work, we aim to further compress trainable parameters by enjoying the powerful expressiveness of the Fourier transform. Specifically, we introduce FourierFT, which treats $ΔW$ as a matrix in the spatial domain and learns only a small fraction of its spectral coefficients. With the trained spectral coefficients, we implement the inverse discrete Fourier transform to recover $ΔW$. Empirically, our FourierFT method shows comparable or better performance with fewer parameters than LoRA on various tasks, including natural language understanding, natural language generation, instruction tuning, and image classification. For example, when performing instruction tuning on the LLaMA2-7B model, FourierFT surpasses LoRA with only 0.064M trainable parameters, compared to LoRA's 33.5M. Our code is released at \url{https://github.com/Chaos96/fourierft}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。