发表机构
Technical University of Munich（慕尼黑工业大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究LoRA微调中入侵者维度出现的问题，通过矩形尖峰变形变换推导出临界更新强度等，在多模型研究中该定律能有效定位阈值、区分有无入侵者层等，解决了LoRA与全量微调的不对称性，还减少了模型遗忘。
AI 中文摘要
LoRA微调会产生入侵者维度，即更新权重矩阵 \(W + BA\) 的新主导奇异向量，它们几乎与所有预训练奇异向量正交并导致灾难性遗忘。自发现以来，尚无理论能逐层层预测其出现的时间。本文通过矩形尖峰变形变换，仅根据 \(W\) 的测量谱推导出每层的临界更新强度 \(s^\ast=\bar\theta/(\gamma\sigma_1(BA))\) 以及更新谱的精确久期方程表征，无拟合参数。在涵盖四个密集Transformer家族、一个状态空间模型、一个专家混合模型和一个编码器 - 解码器（18个适配器，9840层扫描）的预指定研究中，该定律在82%的层上将经验阈值定位在两倍范围内，在部署时以平均AUC为0.89区分有入侵者层和无入侵者层，在六个第三方适配器上保持不变，并预测WikiText - 2困惑度开始下降的位置；两种预指定边缘评估的组合达到98%，并在外部适配器上通过袋外验证得到确认（0.997）。全量微调将其更新分散到远低于每层阈值的水平，解决了LoRA和全量微调之间的不对称性。范数匹配干预证实，阈值交叉层而非更新幅度导致遗忘，从阈值导出的尖峰预算规则在不增加任务成本且无需验证扫描的情况下，将最脆弱模型上的遗忘减少了62%。
英文摘要
LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting. Since their discovery, no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength $s^\ast=\barθ/(γσ_1(BA))$, computed from the measured spectrum of $W$ alone through the rectangular spiked-deformation transform, together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study spanning four dense Transformer families, a state-space model, a mixture-of-experts model, and an encoder-decoder (18 adapters, 9{,}840 layer scans), the law localizes the empirical threshold within a factor of two on $82\%$ of layers, separates intruder-bearing from intruder-free layers at deployment with a mean AUC of $0.89$, holds unchanged on six third-party adapters, and predicts where WikiText-2 perplexity begins to degrade; a combination of the two pre-specified edge evaluations reaches $98\%$ and is confirmed out-of-bag on the external adapters ($0.997$). Full fine-tuning disperses its update far below the threshold of every layer, which resolves the asymmetry between LoRA and full fine-tuning. Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting, and a spike-budget rule derived from the thresholds, requiring one SVD and no validation sweeps, reduces forgetting by $62\%$ on the most fragile model at no task cost.
