arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.02432cs.CL
收藏

Swift跨数据集剪枝:提升自然语言理解中的微调效率

Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding

Agency for Science, Technology and Research(新加坡科技研究局);VNU University of Engineering and Technology(越南国家大学工程技术大学)

Binh-Nguyen Nguyen, Yang He

AI总结:

针对跨数据集任务微调中样本排序成本高的问题,本文提出SCDP,利用TF-IDF嵌入与几何中位数评估样本重要性,并按数据集规模自适应剪枝,在六个数据集上以更低计算成本取得良好效果。

摘要:

数据集剪枝旨在选择数据集的一个子集,以实现高效的模型训练。尽管自然语言处理中的数据效率主要集中在模型预训练期间的语料库内场景,但由于数据集规模、数据分布、类别不平衡和标签空间的差异,面向多样化数据集的任务特定微调的高效数据集剪枝仍然具有挑战性。当前用于微调的跨数据集剪枝技术通常依赖计算成本高昂的样本排序过程,往往需要完整数据集训练或参考模型。我们通过提出Swift跨数据集剪枝(SCDP)来解决这一空白。具体而言,该方法使用TF-IDF嵌入和几何中位数来快速评估样本重要性。随后,我们采用数据集规模自适应剪枝以确保多样性:对于较小的数据集,保留远离几何中位数的样本;对于较大的数据集,则采用基于距离的分层剪枝。在六个多样化数据集上的实验结果证明了该方法的有效性,覆盖多种任务和规模,同时显著减少了计算资源。源代码可在https://github.com/he-y/NLP-Dataset-Pruning获取。

英文摘要:

Dataset pruning aims to select a subset of a dataset for efficient model training. While data efficiency in natural language processing has primarily focused on within-corpus scenarios during model pre-training, efficient dataset pruning for task-specific fine-tuning across diverse datasets remains challenging due to variability in dataset sizes, data distributions, class imbalance and label spaces. Current cross-dataset pruning techniques for fine-tuning often rely on computationally expensive sample ranking processes, typically requiring full dataset training or reference models. We address this gap by proposing Swift Cross-Dataset Pruning (SCDP). Specifically, our approach uses TF-IDF embeddings with geometric median to rapidly evaluate sample importance. We then apply dataset size-adaptive pruning to ensure diversity: for smaller datasets, we retain samples far from the geometric median, while for larger ones, we employ distance-based stratified pruning. Experimental results on six diverse datasets demonstrate the effectiveness of our method, spanning various tasks and scales while significantly reducing computational resources. Source code is available at: https://github.com/he-y/NLP-Dataset-Pruning

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑