arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19458cs.CLcs.LG
收藏

来自多个数据源的可扩展微调:一种一阶近似方法

Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach

Northeastern University, Boston, MA(东北大学);University of Michigan, Ann Arbor, MI(密歇根大学)

Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang R. Zhang

AI总结:

本文提出一种基于一阶近似的算法,利用多任务元初始化的函数值和梯度快速估计辅助任务子集的微调性能,无需反复训练,在指令微调和思维链微调中实现30倍加速且误差仅1%,性能最高提升3.8%。

摘要:

我们研究通过最优利用来自 n 个辅助任务的信息来为目标任务微调语言模型(LM)的问题。该问题在自然语言处理(NLP)中具有广泛应用,例如定向指令微调和思维链微调中的数据选择。该问题的关键挑战在于并非所有辅助任务都有助于提升目标任务的性能,因此选择正确的辅助任务子集至关重要。传统子集选择方法(如前向和后向逐步选择)不适用于 LM 微调,因为它们需要在辅助任务子集上反复训练。本文提出一种新算法,无需反复训练即可估计模型微调性能。我们的算法首先使用所有任务的数据进行多任务训练,以获得元初始化。然后,我们利用元初始化处的函数值和梯度来近似某个子集的模型微调损失。实验上,我们发现这种基于梯度的近似在十二个基于 Transformer 的 LM 上具有显著的准确性。因此,我们现在可以在 CPU 上几秒内估计微调性能。最后,我们在选定的任务子集上对预训练基础模型进行一次微调。我们进行了大量实验来验证该方法,与传统子集选择相比实现了 30 倍的加速,同时仅产生真实微调性能 1% 的误差。在涉及指令微调和思维链微调的下游评估中,这种基于损失的选择方法相比先前基于梯度或表示相似性的子集选择方法最多提升 3.8%。

英文摘要:

We study the problem of fine-tuning a language model (LM) for a target task by optimally using the information from $n$ auxiliary tasks. This problem has broad applications in NLP, such as targeted instruction tuning and data selection in chain-of-thought fine-tuning. The key challenge of this problem is that not all auxiliary tasks are beneficial in improving the performance of the target task. Thus, selecting the right subset of auxiliary tasks is crucial. Conventional subset selection methods, such as forward and backward stepwise selection, are unsuitable for LM fine-tuning because they require repeated training on subsets of auxiliary tasks. This paper introduces a new algorithm for estimating model fine-tuning performance without requiring repeated training. Our algorithm first performs multitask training using data from all tasks to obtain a meta initialization. Then, we approximate the model fine-tuning loss of a subset using functional values and gradients from the meta initialization. Empirically, we find that this gradient-based approximation holds with remarkable accuracy for twelve transformer-based LMs. Thus, we can now estimate fine-tuning performances on CPUs within a few seconds. Finally, we fine-tune the pretrained base model once on the selected subset of tasks. We conduct extensive experiments to validate this approach, delivering a speedup of $30\times$ over conventional subset selection while incurring only $1\%$ error of the true fine-tuning performances. In downstream evaluations involving both instruction tuning and chain-of-thought fine-tuning, this loss-based selection approach improves over prior gradient or representation similarity-based methods for subset selection by up to $3.8\%$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑