arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.18124cs.CLcs.AI
收藏

TLoRA:面向任务的大型语言模型低秩适应

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

College of Computer Science and Software Engineering, Shenzhen University, China(深圳大学计算机科学与软件工程学院,中国)

Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang

AI总结:

TLoRA通过联合优化初始化和资源分配,提升LoRA在不同任务中的性能,减少可训练参数数量。

摘要:

低秩适应(LoRA)已成为大型语言模型参数高效微调的常用方法,其效果受秩分配、缩放因子和初始化的影响。本文提出面向任务的低秩适应(TLoRA),统一框架在训练初期联合优化初始化和资源分配。TLoRA引入数据驱动的初始化策略,通过预训练权重与输入激活协方差的乘积进行奇异值分解,对齐LoRA A矩阵与任务相关子空间。随后冻结A矩阵,仅训练B矩阵。此外,TLoRA采用敏感度基于的重要性度量,在固定参数预算下适应性地分配各层的秩和缩放因子。实验表明,TLoRA在自然语言理解、常识推理、数学推理、代码生成和聊天生成等任务中均表现优异,同时显著减少可训练参数数量。

英文摘要:

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA $A$ matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the $A$ matrix is frozen, and only the $B$ matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑