arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.05356cs.LG
收藏

探索学习复杂度以实现高效下游数据集剪枝

Exploring Learning Complexity for Efficient Downstream Dataset Pruning

Department of Statistics and Data Science, Southern University of Science and Technology(统计与数据科学系,南方科技大学);State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学)

Wenyu Jiang, Zhenlong Liu, Zejian Xie, Songxin Zhang, Bingyi Jing, Hongxin Wei

AI总结:

本文提出无需训练的难度分数DLC,通过轻量级权重掩码快速估计样本学习复杂度,并结合FlexRand欠采样策略,在图像与指令数据集剪枝中实现高效且先进的性能。

摘要:

大规模预训练模型日益增长的微调成本凸显了数据集剪枝的重要性,数据集剪枝旨在减小数据集规模的同时保持任务性能。然而,现有的数据集剪枝方法需要在整个数据集上进行训练,这对于大规模预训练模型而言并不现实。在本文中,我们提出了一种直接、新颖且无需训练的难度分数,称为基于失真的学习复杂度(Distorting-based Learning Complexity,DLC),用于高效地从下游数据集中识别有信息量的图像和指令。我们的方法源于以下观察:学习速度更快的简单样本也可以用更少的参数学习。具体而言,我们定义学习复杂度来量化样本难度,并利用轻量级权重掩码过程进行快速估计,以替代代价高昂的随机梯度下降(SGD)优化。基于DLC,我们进一步设计了一种带随机性的灵活欠采样方法(称为FlexRand),取代top-K策略,以缓解严重的子集分布偏移。在下游图像和指令数据集剪枝基准上的大量实验证明了所提方法的有效性和高效性。在图像剪枝基准中,DLC将剪枝时间显著减少了35倍,同时借助FlexRand建立了最先进的性能。

英文摘要:

The ever-increasing fine-tuning cost of large-scale pre-trained models gives rise to the importance of dataset pruning, which aims to reduce dataset size while maintaining task performance. However, existing dataset pruning methods require training on the entire dataset, which is impractical for large-scale pre-trained models. In this paper, we propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC), to identify informative images and instructions from the downstream dataset efficiently. Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters. Specifically, we define the Learning Complexity to quantify sample hardness and utilize a lightweight weights masking process for fast estimation, instead of the costly SGD optimization. Based on DLC, we further design a flexible under-sampling with randomness (dubbed FlexRand), replacing the top-K strategy, to alleviate the severe subset distribution shift. Extensive experiments with downstream image and instructions dataset pruning benchmarks demonstrate the effectiveness and efficiency of the proposed approach. In the images pruning benchmark, DLC significantly reduces the pruning time by 35x while establishing state-of-the-art performance with FlexRand.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑