GRAD-MATCH:基于梯度匹配的数据子集选择,用于有效的深度模型训练
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
摘要:
现代机器学习模型在大型数据集上的巨大成功取决于大量的计算资源,以及高昂的财务和环境成本。解决这个问题的一种方法是提取与完整数据同等的子集。在这项工作中,我们提出了一个通用的框架,梯度匹配,它发现的子集,密切匹配的梯度的训练或验证集。我们发现这样的子集有效地使用正交匹配追踪算法。我们展示了所提出的算法的严格的理论和收敛保证,并通过我们在真实世界数据集上的广泛实验,展示了我们所提出的框架的有效性。我们表明,GRAD-MATCH显着和一贯优于最近的几个数据选择算法,并实现了最佳的准确性和效率的权衡。GRAD-MATCH是CORDS工具包\url{https://github.com/decile-team/cords}的一部分。
英文摘要:
The great success of modern machine learning models on large datasets is contingent on extensive computational resources with high financial and environmental costs. One way to address this is by extracting subsets that generalize on par with the full data. In this work, we propose a general framework, GRAD-MATCH, which finds subsets that closely match the gradient of the training or validation set. We find such subsets effectively using an orthogonal matching pursuit algorithm. We show rigorous theoretical and convergence guarantees of the proposed algorithm and, through our extensive experiments on real-world datasets, show the effectiveness of our proposed framework. We show that GRAD-MATCH significantly and consistently outperforms several recent data-selection algorithms and achieves the best accuracy-efficiency trade-off. GRAD-MATCH is available as a part of the CORDS toolkit: \url{https://github.com/decile-team/cords}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。