arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.06708cs.LGcs.AI
收藏

通过模仿模型权重评估样本效用以实现高效数据选择

Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

Apple(苹果公司)

Tzu-Heng Huang, Manjot Bilkhu, John Cooper, Frederic Sala, Javier Movellan

AI总结:

提出基于梯度和几何的Mimic Score指标,通过Grad-Mimic框架在线重加权样本加速训练、离线构建数据过滤器,在六个图像数据集上提升数据效率和CLIP模型性能。

摘要:

大规模网络爬取数据集包含噪声、偏差和不相关信息,因此需要数据选择技术。现有方法依赖于手工启发式、下游数据集或需要昂贵的基于影响力的计算——所有这些都限制了可扩展性并引入了不必要的数据依赖性。为了解决这个问题,我们引入了Mimic Score,一种简单且基于几何的数据质量指标,通过测量样本梯度与预训练参考模型诱导的目标方向之间的对齐来评估效用。这利用了现成的模型权重,避免了验证数据集的需求,并且计算开销最小。基于该指标,我们提出了Grad-Mimic,一个两阶段框架,在线重新加权样本以加速训练,并离线聚合样本效用以构建有效的数据过滤器。实验表明,使用模仿分数指导训练提高了数据效率,加速了收敛,在六个图像数据集上取得了一致的性能提升,并以减少20.7%的训练步骤增强了CLIP模型。此外,基于模仿分数的过滤器增强了现有过滤技术,使得用更少470万个样本训练的CLIP模型得到改进。

英文摘要:

Large-scale web-crawled datasets contain noise, bias, and irrelevant information, necessitating data selection techniques. Existing methods depend on hand-crafted heuristics, downstream datasets, or require expensive influence-based computations -- all of which limit scalability and introduce unwanted data dependencies. To address this, we introduce the Mimic Score, a simple and geometry-based data-quality metric that evaluates utility by measuring alignment between a sample's gradients and a target direction induced by a pre-trained reference model. This leverages readily available model weights, avoids needing validation datasets, and incurs minimal computational overheads. Building on this metric, we propose Grad-Mimic, a two-stage framework that re-weights samples online to accelerate training and aggregates sample utilities offline to construct effective data filters. Empirically, we show that using mimic scores to guide training improves data efficiency, accelerates convergence, yields consistent performance gains across six image datasets, and enhances CLIP models with 20.7% fewer training steps. Additionally, mimic score-based filters augment existing filtering techniques, enabling improved CLIP models trained with 4.7 million fewer samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑