arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.10630cs.LGcs.AI
收藏

GLISTER:基于泛化的数据子集选择算法

GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning

University of Texas at Dallas(德克萨斯大学达拉斯分校);Indian Institute of Technology, Bombay(印度理工学院孟买分校)

Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Rishabh Iyer

摘要:

大规模机器学习和深度模型非常需要数据。不幸的是,获得大量的标记数据是昂贵的,并且训练最先进的模型(具有超参数调整)需要大量的计算资源和时间。其次,真实世界的数据是嘈杂和不平衡的。因此,最近的几篇论文试图使训练过程更有效和鲁棒。然而,大多数现有的工作要么集中在鲁棒性或效率,但不是两者兼而有之。在这项工作中,我们介绍了Glister,一个基于泛化的数据子集选择的高效和鲁棒的学习框架。我们制定Glister作为一个混合的离散-连续的双层优化问题,以选择一个子集的训练数据,最大限度地提高了对数似然验证集。接下来,我们提出了一种迭代在线算法Glister-Online,它随着参数更新迭代地执行数据选择,并且可以应用于任何基于损失的学习算法。然后,我们表明,丰富的一类损失函数,包括交叉熵,铰链损失,平方损失,和物流损失,内部离散数据的选择是一个实例(弱)子模块优化,我们分析的条件下,Glister在线减少验证损失和收敛。最后,我们提出了Glister-Active,批量主动学习的扩展,我们经验证明了Glister在各种任务上的性能,包括:(a)数据选择,以减少训练时间,(b)标签噪声和不平衡设置下的鲁棒学习,以及(c)批量主动学习与几个深度和浅模型。我们表明,我们的框架在效率和准确性方面都提高了现有技术(在情况(a)和(c)中),并且与其他最先进的鲁棒学习算法相比,在情况(b)中更有效。

英文摘要:

Large scale machine learning and deep models are extremely data-hungry. Unfortunately, obtaining large amounts of labeled data is expensive, and training state-of-the-art models (with hyperparameter tuning) requires significant computing resources and time. Secondly, real-world data is noisy and imbalanced. As a result, several recent papers try to make the training process more efficient and robust. However, most existing work either focuses on robustness or efficiency, but not both. In this work, we introduce Glister, a GeneraLIzation based data Subset selecTion for Efficient and Robust learning framework. We formulate Glister as a mixed discrete-continuous bi-level optimization problem to select a subset of the training data, which maximizes the log-likelihood on a held-out validation set. Next, we propose an iterative online algorithm Glister-Online, which performs data selection iteratively along with the parameter updates and can be applied to any loss-based learning algorithm. We then show that for a rich class of loss functions including cross-entropy, hinge-loss, squared-loss, and logistic-loss, the inner discrete data selection is an instance of (weakly) submodular optimization, and we analyze conditions for which Glister-Online reduces the validation loss and converges. Finally, we propose Glister-Active, an extension to batch active learning, and we empirically demonstrate the performance of Glister on a wide range of tasks including, (a) data selection to reduce training time, (b) robust learning under label noise and imbalance settings, and (c) batch-active learning with several deep and shallow models. We show that our framework improves upon state of the art both in efficiency and accuracy (in cases (a) and (c)) and is more efficient compared to other state-of-the-art robust learning algorithms in case (b).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑