arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.17293cs.LGcs.AI
收藏

高效集成提升训练数据归因

Efficient Ensembles Improve Training Data Attribution

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Junwei Deng, Ting-Wei Li, Shichang Zhang, Jiaqi Ma

AI总结:

本研究提出两种高效集成策略(DROPOUT ENSEMBLE和LORA ENSEMBLE),在显著降低训练、服务及空间成本的同时,保持与独立集成相当的训练数据归因效果,推进了TDA方法的帕累托前沿。

摘要:

训练数据归因(TDA)方法旨在量化单个训练数据点对模型预测的影响,在数据为中心的AI中具有广泛应用,如错误标签检测、数据选择和版权补偿。然而,该领域的现有方法可分为基于重训练和基于梯度两类,它们在计算效率与归因效果之间难以权衡。基于重训练的方法能准确归因复杂的非凸模型,但计算代价高昂;而基于梯度的方法虽高效,但往往在非凸模型上失效。近期研究表明,将基于梯度的方法与多个独立训练模型的集成相结合,可显著提升归因效果。然而,这种方法在超大规模应用中仍不切实际。在本工作中,我们发现,对于集成基于梯度的方法,昂贵的完全独立训练并非必要,并提出了两种高效集成策略——DROPOUT ENSEMBLE和LORA ENSEMBLE,作为朴素独立集成的替代方案。这些策略显著减少了训练时间(最高达80%)、服务时间(最高达60%)和空间成本(最高达80%),同时保持与朴素独立集成相当的归因效果。我们的大量实验结果表明,所提策略在多种TDA方法、多样数据集和模型(包括生成设置)中均有效,显著推进了TDA方法的帕累托前沿,实现了更好的计算效率和归因效果。

英文摘要:

Training data attribution (TDA) methods aim to quantify the influence of individual training data points on the model predictions, with broad applications in data-centric AI, such as mislabel detection, data selection, and copyright compensation. However, existing methods in this field, which can be categorized as retraining-based and gradient-based, have struggled with the trade-off between computational efficiency and attribution efficacy. Retraining-based methods can accurately attribute complex non-convex models but are computationally prohibitive, while gradient-based methods are efficient but often fail for non-convex models. Recent research has shown that augmenting gradient-based methods with ensembles of multiple independently trained models can achieve significantly better attribution efficacy. However, this approach remains impractical for very large-scale applications. In this work, we discover that expensive, fully independent training is unnecessary for ensembling the gradient-based methods, and we propose two efficient ensemble strategies, DROPOUT ENSEMBLE and LORA ENSEMBLE, alternative to naive independent ensemble. These strategies significantly reduce training time (up to 80%), serving time (up to 60%), and space cost (up to 80%) while maintaining similar attribution efficacy to the naive independent ensemble. Our extensive experimental results demonstrate that the proposed strategies are effective across multiple TDA methods on diverse datasets and models, including generative settings, significantly advancing the Pareto frontier of TDA methods with better computational efficiency and attribution efficacy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑