ReCoLoRA：用于持续语言模型微调的频谱感知递归合并
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
AI总结：
研究针对持续语言模型微调中类似LoRA方法新任务覆盖旧任务的问题，提出ReCoLoRA框架，通过特定初始化、选秩及递归合并等操作，在多任务持续GLUE序列实验中，用更少参数取得更好成绩。
摘要：
参数高效微调可低成本地使大语言模型适应一项任务，但在跨任务序列时，类似LoRA的方法会在相同的冻结权重上不断叠加低秩更新，导致新任务倾向于覆盖先前任务。我们提出了ReCoLoRA（低秩适配器的递归合并），这是一种用于持续微调的频谱感知框架：适配器从预训练权重的随机奇异值分解初始化，通过肘部准则选择每层的有效秩，并在开放剩余容量之前调整主子空间。在每个新任务之前，ReCoLoRA将当前有效权重而非原始权重重新分解为冻结残差、缓慢更新的主成分和新的适配器（递归合并），使每个任务都从已吸收其前身的模型开始。在四个包含7 - 8B参数的主干上进行的六任务持续GLUE序列实验中，与LoRA、PiSSA、AdaLoRA和DoRA基线相比，ReCoLoRA在四个主干中的三个上获得了最佳最终平均分数，同时训练的参数更少；一个神谕路由任务库变体在完全任务隔离下作为上限。代码：此https URL。
英文摘要：
Parameter-efficient fine-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA-style methods keep stacking low-rank updates on the same frozen weight, so each new task tends to overwrite the previous ones. We present ReCoLoRA (Recursive Consolidation of Low-Rank Adapters), a spectrum-aware framework for continual fine-tuning: adapters are initialized from a randomized SVD of the pretrained weight, per-layer effective ranks are selected by an elbow criterion, and the principal subspace is adapted before residual capacity is opened. Before each new task, ReCoLoRA re-decomposes the current effective weight, rather than the original one, into a frozen residual, a slowly updated principal component, and a fresh adapter (recursive consolidation), so every task starts from the model that has already absorbed its predecessors. On a six-task continual GLUE sequence over four 7-8B backbones, ReCoLoRA attains the best final average score on three of the four backbones against rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while training fewer parameters; an oracle-routed task-bank variant serves as an upper bound under full task isolation. Code: https://github.com/bhqy666/ReCoLoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
