arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.22337cs.LGcs.AI
收藏

近似正则化回放的有效性:用于大型语言模型高效监督微调的效果

The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models

IBM Research(IBM研究院);Mila, Université de Montréal(蒙特利尔大学Mila)

Matthew Riemer, Erik Miehling, Miao Liu, Djallel Bouneffouf, Murray Campbell

AI总结:

本文提出了一种正则化近似回放方法,通过惩罚KL散度和引入不同语料库数据,有效缓解LoRA监督微调导致的模型能力下降问题。

摘要:

尽管参数高效微调方法,如LoRA,只修改一小部分参数,但它们对模型有显著影响。我们的指令微调实验表明,基于LoRA的监督微调可能会在非常小的数据集上经过相对少量步骤训练后,严重降低模型能力。然而,我们证明,尽管最直接的方法(可能是实践中最常用的)表现糟糕,但通过很小的训练过程调整,几乎可以消除这个问题。特别是,在本文中,我们考虑了一种正则化近似回放方法,该方法通过惩罚与初始模型的KL散度,并在下一个token预测中插入来自不同但相似的开放访问语料库的数据。当应用于Qwen指令微调模型时,我们发现该方法在不阻碍对新任务的适应性的情况下,能够保持模型的一般知识,并通过增加少量计算开销实现。

英文摘要:

Although parameter-efficient fine-tuning methods, such as LoRA, only modify a small subset of parameters, they can have a significant impact on the model. Our instruction-tuning experiments show that LoRA-based supervised fine-tuning can catastrophically degrade model capabilities, even when trained on very small datasets for relatively few steps. With that said, we demonstrate that while the most straightforward approach (that is likely the most used in practice) fails spectacularly, small tweaks to the training procedure with very little overhead can virtually eliminate the problem. Particularly, in this paper we consider a regularized approximate replay approach which penalizes KL divergence with respect to the initial model and interleaves in data for next token prediction from a different, yet similar, open access corpus to what was used in pre-training. When applied to Qwen instruction-tuned models, we find that this recipe preserves general knowledge in the model without hindering plasticity to new tasks by adding a modest amount of computational overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑