Perplexity 之惑:基于 Perplexity 的小参考模型数据剪枝
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
AI总结:
本文研究了使用小型语言模型基于 perplexity 进行数据剪枝的有效性,发现其能显著提升大模型下游任务性能并减少预训练步骤,且在过训练和数据受限场景下同样有效。
摘要:
在这项工作中,我们研究了小型语言模型是否能够确定大规模文本数据集的高质量子集,从而提升更大语言模型的性能。尽管现有工作表明基于更大模型的 perplexity 进行剪枝可以产生高质量数据,但我们研究了是否可以使用更小的模型进行基于 perplexity 的剪枝,以及剪枝如何受到被剪枝数据的领域组成的影响。我们证明,对于多种数据集组成,基于 perplexity 的预训练数据剪枝能够显著提升下游任务性能:使用 1.25 亿参数模型计算 perplexity 进行剪枝,使 30 亿参数模型在下游任务上的平均性能提升了 2.04,并使达到同等基线性能所需的预训练步骤减少了 1.45 倍。此外,我们还证明这种基于 perplexity 的数据剪枝在过训练和数据受限的情况下同样能带来下游性能的提升。
英文摘要:
In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of a larger model can yield high-quality data, we investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-based pruning of pretraining data can \emph{significantly} improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of a 3 billion parameter model by up to 2.04 and achieves up to a $1.45\times$ reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity-based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。