arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.04060cs.CLcs.AI
收藏

LoRTA:大语言模型的低秩张量适应

LoRTA: Low Rank Tensor Adaptation of Large Language Models

University of YYY(YYY大学);Institute of WWW(WWW研究院)

Ignacio Hounie, Charilaos Kanatsoulis, Arnuv Tandon, Alejandro Ribeiro

AI总结:

本文提出LoRTA,一种基于高阶Candecomp/Parafac(CP)分解的低秩张量适应方法,通过更紧凑灵活的表示减少可训练参数数量,并在多项基准上保持可比性能。

摘要:

低秩适应(LoRA)是一种流行的参数高效微调(PEFT)方法,能有效适应大型预训练模型以用于下游任务。LoRA在每一层使用低秩矩阵对模型更新进行参数化,显著减少了可训练参数数量,从而降低了微调过程中的资源需求。然而,由于使用低秩矩阵模型,可训练参数数量的下限仍然较高。近期的研究通过提出用于模型更新的低秩张量参数化来解决这一限制。然而,它们仅利用跨层冗余,或使用引入额外超参数的临时方案对单个矩阵进行张量化。在本研究中,我们提出了一种高阶Candecomp/Parafac(CP)分解,与现有的基于矩阵和张量的PEFT方法相比,能够实现更紧凑且灵活的表示。我们在自然语言理解、指令微调、偏好优化和蛋白质折叠基准上的实验表明,我们的方法能够在保持可比性能的同时减少参数数量。

英文摘要:

Low Rank Adaptation (LoRA) is a popular Parameter Efficient Fine Tuning (PEFT) method that effectively adapts large pre-trained models for downstream tasks. LoRA parameterizes model updates using low-rank matrices at each layer, significantly reducing the number of trainable parameters and, consequently, resource requirements during fine-tuning. However, the lower bound on the number of trainable parameters remains high due to the use of the low-rank matrix model. Recent works have addressed this limitation by proposing low rank tensor parameterizations for model updates. However, they only exploit redundancy across layers, or tensorize individual matrices using ad-hoc schemes that introduce additional hyperparameters. In this work, we propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation compared to existing matrix and tensor based PEFT methods. Our experiments on Natural Language Understanding, Instruction Tuning, Preference Optimization and Protein Folding benchmarks demonstrate that our method can achieve a reduction in the number of parameters while maintaining comparable performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑