arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.08582cs.CL
收藏

LoRS: 面向稀疏大语言模型的高效低秩适配

LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model

School of Information, Renmin University of China(中国人民大学信息学院);Tencent AI Lab(腾讯AI实验室)

Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen

AI总结:

针对现有低秩适配方法在稀疏大语言模型上难以兼顾稀疏性保持与效率的问题,提出LoRS方法,通过权重重计算、计算图重排及优化适配器初始化,实现微调阶段内存与计算消耗降低且性能更优。

摘要:

现有的低秩适配(LoRA)方法在稀疏大语言模型(LLM)上面临挑战,原因是其无法保持稀疏性。近期研究通过在LoRA技术中加入额外掩码机制,提出了可保持稀疏性的方法。尽管取得了一定成效,但这类方法的内存与计算开销有所增加,影响了LoRA方法的效率。针对这一局限,我们提出LoRS——一种创新方法,旨在实现稀疏LLM微调时的内存与计算双高效。为缓解保持稀疏性带来的高额内存与计算需求,我们的方法采用了权重重计算与计算图重排策略。此外,我们还通过更优的适配器初始化提升了LoRS的效果。这些创新显著降低了微调阶段的内存与计算消耗,同时性能水平优于现有LoRA方法。

英文摘要:

Existing low-rank adaptation (LoRA) methods face challenges on sparse large language models (LLMs) due to the inability to maintain sparsity. Recent works introduced methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects efficiency of LoRA methods. In response to this limitation, we introduce LoRS, an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recompute and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, all while achieving performance levels that outperform existing LoRA approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑