arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.03107cs.LGcs.CVeess.IVstat.ML
收藏

课程什么时候起作用?

When Do Curricula Work?

UT Austin(德克萨斯大学奥斯汀分校);Google(谷歌)

Xiaoxia Wu, Ethan Dyer, Behnam Neyshabur

摘要:

受人类学习的启发,研究人员提出了在训练过程中根据难度对示例进行排序。课程学习和反课程学习都被认为是对标准i.i.d.的改进,前者在训练早期将网络暴露于较容易的示例,后者首先显示最困难的示例。训练在这项工作中,我们开始研究有序学习的相对好处。我们首先调查的\emph{implicit curricula}造成的架构和优化的偏见,并发现,样本的学习在一个高度一致的顺序。接下来,为了量化\emph{explicit curricula}的好处,我们进行了大量的实验,涉及三种学习方式的数千种排序:课程,反课程和随机课程-其中训练数据集的大小随着时间的推移而动态增加,但示例是随机排序的。我们发现,对于标准的基准数据集,课程只有边际效益,随机排序的样本表现得和课程和反课程一样好或更好,这表明任何好处都完全是由于动态训练集的大小。受实践中课程学习的常见用例的启发,我们研究了有限的训练时间预算和噪声数据在课程学习成功中的作用。我们的实验表明,课程,而不是反课程确实可以提高性能,无论是在有限的训练时间预算或存在噪声数据。

英文摘要:

Inspired by human learning, researchers have proposed ordering examples during training based on their difficulty. Both curriculum learning, exposing a network to easier examples early in training, and anti-curriculum learning, showing the most difficult examples first, have been suggested as improvements to the standard i.i.d. training. In this work, we set out to investigate the relative benefits of ordered learning. We first investigate the \emph{implicit curricula} resulting from architectural and optimization bias and find that samples are learned in a highly consistent order. Next, to quantify the benefit of \emph{explicit curricula}, we conduct extensive experiments over thousands of orderings spanning three kinds of learning: curriculum, anti-curriculum, and random-curriculum -- in which the size of the training dataset is dynamically increased over time, but the examples are randomly ordered. We find that for standard benchmark datasets, curricula have only marginal benefits, and that randomly ordered samples perform as well or better than curricula and anti-curricula, suggesting that any benefit is entirely due to the dynamic training set size. Inspired by common use cases of curriculum learning in practice, we investigate the role of limited training time budget and noisy data in the success of curriculum learning. Our experiments demonstrate that curriculum, but not anti-curriculum can indeed improve the performance either with limited training time budget or in existence of noisy data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑