arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.07741cs.LG
收藏

基于损失的优先级什么时候会失败?

When does loss-based prioritization fail?

ML Collective(ML 联盟);Uber(优步);Google Research(谷歌研究院)

Niel Teng Hu, Xinyu Hu, Rosanne Liu, Sara Hooker, Jason Yosinski

摘要:

并不是所有的例子都是平等的,但是标准的深度神经网络训练协议统一对待每个训练点。每个示例通过网络向前和向后传播相同的次数,而与示例对学习协议的贡献大小无关。最近的工作提出了通过偏离这种统一处理来加速训练的方法。流行的方法需要增加权重的例子,这些例子对损失的贡献更大,直觉上,损失低的例子已经被模型学习过了,所以它们对训练过程的边际值应该更低。此视图假定使用高损失示例更新模型将对模型有益。然而,这可能不适用于嘈杂的真实世界数据。在本文中,我们从理论上,然后实证证明,基于损失的加速方法在有噪声和损坏的数据的情况下会降级。我们的工作表明,示例难度的测量需要正确地将噪音从其他类型的具有挑战性的示例中分离出来。

英文摘要:

Not all examples are created equal, but standard deep neural network training protocols treat each training point uniformly. Each example is propagated forward and backward through the network the same amount of times, independent of how much the example contributes to the learning protocol. Recent work has proposed ways to accelerate training by deviating from this uniform treatment. Popular methods entail up-weighting examples that contribute more to the loss with the intuition that examples with low loss have already been learned by the model, so their marginal value to the training procedure should be lower. This view assumes that updating the model with high loss examples will be beneficial to the model. However, this may not hold for noisy, real world data. In this paper, we theorize and then empirically demonstrate that loss-based acceleration methods degrade in scenarios with noisy and corrupted data. Our work suggests measures of example difficulty need to correctly separate out noise from other types of challenging examples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑