arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.03796cs.LG
收藏

课程学习与迁移学习:理论及深度网络实验

Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks

The Hebrew University of Jerusalem(耶路撒冷希伯来大学)

Daphna Weinshall, Gad Cohen, Dan Amir

AI总结:

本文从理论和实验两方面研究课程学习,证明理想课程学习收敛速度随难度递增,并提出基于迁移学习的课程推断方法,在CNN训练中显著提升初期收敛速度并改善泛化性能。

摘要:

我们在优化凸线性回归损失的随机梯度下降背景下,对课程学习进行了理论探究。我们证明了理想课程学习方法的收敛速度随样本难度的增加而单调递增。此外,在所有难度相同的点中,当使用相对于当前假设产生更高损失的样本时,收敛速度更快。随后,我们在训练CNN的背景下分析了课程学习。我们描述了一种方法,该方法通过从另一个在不同任务上预训练的网络进行迁移学习来推断课程。虽然这种方法只能近似理想课程,但我们在经验上观察到与理论预测相似的行为,即训练初期收敛速度显著提升。当任务变得更困难时,也观察到泛化性能的改善。最后,课程学习对不利条件(如过度正则化)表现出鲁棒性。

英文摘要:

We provide theoretical investigation of curriculum learning in the context of stochastic gradient descent when optimizing the convex linear regression loss. We prove that the rate of convergence of an ideal curriculum learning method is monotonically increasing with the difficulty of the examples. Moreover, among all equally difficult points, convergence is faster when using points which incur higher loss with respect to the current hypothesis. We then analyze curriculum learning in the context of training a CNN. We describe a method which infers the curriculum by way of transfer learning from another network, pre-trained on a different task. While this approach can only approximate the ideal curriculum, we observe empirically similar behavior to the one predicted by the theory, namely, a significant boost in convergence speed at the beginning of training. When the task is made more difficult, improvement in generalization performance is also observed. Finally, curriculum learning exhibits robustness against unfavorable conditions such as excessive regularization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑