单个线性层生成任务自适应的低秩矩阵
A Single Linear Layer Yields Task-Adapted Low-Rank Matrices
AI总结:
本研究深入分析了LoRA中初始权重与低秩矩阵的关系,发现各层转换矩阵相似,据此提出CondLoRA方法,利用单个线性层生成任务自适应低秩矩阵,在可训练参数更少的情况下保持与LoRA相当的性能。
摘要:
低秩自适应(Low-Rank Adaptation, LoRA)是一种广泛使用的参数高效微调(Parameter-Efficient Fine-Tuning, PEFT)方法,它通过由两个低秩矩阵 $A$ 和 $B$ 组成的增量矩阵 $ΔW$ 来更新初始权重矩阵 $W_0$。先前的研究表明,$W_0$ 和 $ΔW$ 之间存在相关性。在本研究中,我们旨在深入探讨 $W_0$ 与低秩矩阵 $A$ 和 $B$ 之间的关系,以进一步理解 LoRA 的行为。具体而言,我们分析了一个将 $W_0$ 转换为低秩矩阵的转换矩阵,该矩阵包含了关于这些关系的信息。我们的分析揭示,各层的转换矩阵是相似的。受这些发现的启发,我们假设一个以每层的 $W_0$ 为输入的单个线性层,能够生成任务自适应的低秩矩阵。为了验证这一假设,我们设计了一种名为条件参数化 LoRA(Conditionally Parameterized LoRA, CondLoRA)的方法,该方法使用从单个线性层导出的低秩矩阵来更新初始权重矩阵。我们的实验结果表明,尽管 CondLoRA 的可训练参数少于 LoRA,但其性能仍与 LoRA 不相上下。因此,我们得出结论:“单个线性层能够生成任务自适应的低秩矩阵。”
英文摘要:
Low-Rank Adaptation (LoRA) is a widely used Parameter-Efficient Fine-Tuning (PEFT) method that updates an initial weight matrix $W_0$ with a delta matrix $ΔW$ consisted by two low-rank matrices $A$ and $B$. A previous study suggested that there is correlation between $W_0$ and $ΔW$. In this study, we aim to delve deeper into relationships between $W_0$ and low-rank matrices $A$ and $B$ to further comprehend the behavior of LoRA. In particular, we analyze a conversion matrix that transform $W_0$ into low-rank matrices, which encapsulates information about the relationships. Our analysis reveals that the conversion matrices are similar across each layer. Inspired by these findings, we hypothesize that a single linear layer, which takes each layer's $W_0$ as input, can yield task-adapted low-rank matrices. To confirm this hypothesis, we devise a method named Conditionally Parameterized LoRA (CondLoRA) that updates initial weight matrices with low-rank matrices derived from a single linear layer. Our empirical results show that CondLoRA maintains a performance on par with LoRA, despite the fact that the trainable parameters of CondLoRA are fewer than those of LoRA. Therefore, we conclude that "a single linear layer yields task-adapted low-rank matrices."
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。