AILoRA:面向大型语言模型低秩适配的函数感知非对称初始化
AILoRA: Function-Aware Asymmetric Initialization for Low-Rank Adaptation of Large Language Models
AI总结:
AILoRA提出函数感知非对称初始化方法,利用自注意力中W^Q和W^V的参数差异,分别注入主成分和次要成分,提升LoRA微调性能与收敛速度。
摘要:
参数高效微调(PEFT)旨在缓解将大规模预训练模型适配到多样下游任务时所涉及的巨大计算和内存开销。在众多PEFT策略中,低秩适配(LoRA)因其稳健的经验性能和较低的实现复杂度而成为最广泛采用的方法之一。在实际部署中,LoRA通常应用于自注意力模块的$W^Q$和$W^V$投影矩阵,从而在模型性能和参数效率之间实现有效权衡。尽管LoRA已取得相当大的经验成功,它仍面临次优性能和收敛缓慢等挑战。为解决这些局限,我们引入\textbf{AILoRA},一种融合函数感知非对称低秩先验的新型参数高效方法。我们的经验分析揭示,自注意力机制中的投影矩阵$W^Q$和$W^V$因其功能差异而表现出不同的参数特征。具体而言,$W^Q$捕获对注意力分布计算至关重要的任务特定语义空间知识,使其参数对下游任务变化高度敏感。相比之下,$W^V$编码跨任务和层保持稳定的令牌级特征表示。利用这些见解,AILoRA通过注入$W^Q$的主成分以保留任务自适应能力,以及$W^V$的次要成分以保留可泛化的特征表示,执行函数感知初始化。这种非对称初始化策略使LoRA模块能更好地捕获注意力参数的专业角色,从而提升微调性能和收敛效率。
英文摘要:
Parameter-efficient finetuning (PEFT) aims to mitigate the substantial computational and memory overhead involved in adapting large-scale pretrained models to diverse downstream tasks. Among numerous PEFT strategies, Low-Rank Adaptation (LoRA) has emerged as one of the most widely adopted approaches due to its robust empirical performance and low implementation complexity. In practical deployment, LoRA is typically applied to the $W^Q$ and $W^V$ projection matrices of self-attention modules, enabling an effective trade-off between model performance and parameter efficiency. While LoRA has achieved considerable empirical success, it still encounters challenges such as suboptimal performance and slow convergence. To address these limitations, we introduce \textbf{AILoRA}, a novel parameter-efficient method that incorporates function-aware asymmetric low-rank priors. Our empirical analysis reveals that the projection matrices $W^Q$ and $W^V$ in the self-attention mechanism exhibit distinct parameter characteristics, stemming from their functional differences. Specifically, $W^Q$ captures task-specific semantic space knowledge essential for attention distributions computation, making its parameters highly sensitive to downstream task variations. In contrast, $W^V$ encodes token-level feature representations that tend to remain stable across tasks and layers. Leveraging these insights, AILoRA performs a function-aware initialization by injecting the principal components of $W^Q$ to retain task-adaptive capacity, and the minor components of $W^V$ to preserve generalizable feature representations. This asymmetric initialization strategy enables LoRA modules to better capture the specialized roles of attention parameters, thereby enhancing both finetuning performance and convergence efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。