SmallToLarge (S2L):通过总结小模型的训练轨迹实现可扩展的大语言模型微调数据选择
SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
AI总结:
本文提出 SmallToLarge(S2L)数据选择方法,利用小模型训练轨迹指导大模型 SFT,在数学推理和临床摘要任务中以少量数据达到或超过全量训练性能,并降低选择成本。
摘要:
尽管数据选择在大语言模型(LLM)的预训练和指令微调阶段已显示出有效性,但由于微调数据的复杂性,在专业领域的监督微调(SFT)中提升数据效率仍面临重大挑战。为弥合这一差距,我们提出了一种有效且可扩展的 SFT 数据选择方法 SmallToLarge(S2L),该方法利用小模型的训练轨迹来指导更大模型的数据选择。通过大量实验,我们证明 S2L 显著提升了数学问题求解 SFT 的数据效率:仅使用原始 MathInstruct 数据集(Yue 等人,2023)11% 的训练数据即可达到完整数据集的性能,同时在 6 个域内和域外评估数据集上平均比最先进的数据选择算法高出 4.7%。值得注意的是,仅选择 50K 条数据进行 SFT,S2L 就在最具挑战性的 MATH(Hendrycks 等人,2021)基准上取得 32.7% 的准确率,使 Phi-2(Li 等人,2023b)提升 16.6%。在 MIMIC-III 数据集(Johnson 等人,2016)的临床文本摘要任务中,S2L 仅使用 50% 的数据便再次超过使用完整数据集训练的效果。此外,S2L 能够使用比目标模型小 40 倍的参考模型执行数据选择,从而按比例降低数据选择成本。
英文摘要:
Despite the effectiveness of data selection for large language models (LLMs) during pretraining and instruction fine-tuning phases, improving data efficiency in supervised fine-tuning (SFT) for specialized domains poses significant challenges due to the complexity of fine-tuning data. To bridge this gap, we introduce an effective and scalable data selection method for SFT, SmallToLarge (S2L), which leverages training trajectories from small models to guide the data selection for larger models. We demonstrate through extensive experiments that S2L significantly improves data efficiency in SFT for mathematical problem-solving, reducing the training data to just 11% of the original MathInstruct dataset (Yue et al., 2023) to match full dataset performance while outperforming state-of-the-art data selection algorithms by an average of 4.7% across 6 in- and out-domain evaluation datasets. Remarkably, selecting only 50K data for SFT, S2L achieves a 32.7% accuracy on the most challenging MATH (Hendrycks et al., 2021) benchmark, improving Phi-2 (Li et al., 2023b) by 16.6%. In clinical text summarization on the MIMIC-III dataset (Johnson et al., 2016), S2L again outperforms training on the full dataset using only 50% of the data. Notably, S2L can perform data selection using a reference model 40x smaller than the target model, proportionally reducing the cost of data selection.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。