arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1611.03068cs.LGcs.NE
收藏

增量序列学习

Incremental Sequence Learning

Utrecht University(乌得勒支大学)

Edwin D. de Jong

摘要:

过去几年的深度学习研究表明,随着时间的推移,通过增加学习问题的范围或难度,可以解决日益复杂的学习问题。我们在序列学习的背景下研究增量学习,使用多层递归混合密度网络形式的生成RNN。虽然渐进式学习或课程学习在促进学习方面的潜力是众所周知的,但不加区别地应用这一原则并不一定会导致改进,因此,必须知道哪些形式的渐进式学习或课程学习具有积极的效果。本研究通过比较三个渐进式或课程学习的实例,有助于实现这一目标。我们介绍增量序列学习,一个简单的增量序列学习方法。增量序列学习开始时仅使用每个序列的前几步作为训练数据。每次达到性能标准时,用于训练的序列部分的长度增加。我们介绍并提供一种新的序列学习任务和数据集:预测和分类MNIST笔画序列。我们发现,增量序列学习大大加快了序列学习的速度,并以20倍的速度达到常规序列学习的最佳测试性能水平,将测试误差降低了74%,并且通常表现得更稳健;它显示出较低的方差,并在所有三种比较方法停止改进后取得了持续的进展。课程学习的其他实例并没有带来任何明显的改善。训练的序列预测模型也用于序列分类任务的迁移学习,其中发现与从头开始学习分类的方法相比,迁移学习实现了改进的分类性能。

英文摘要:

Deep learning research over the past years has shown that by increasing the scope or difficulty of the learning problem over time, increasingly complex learning problems can be addressed. We study incremental learning in the context of sequence learning, using generative RNNs in the form of multi-layer recurrent Mixture Density Networks. While the potential of incremental or curriculum learning to enhance learning is known, indiscriminate application of the principle does not necessarily lead to improvement, and it is essential therefore to know which forms of incremental or curriculum learning have a positive effect. This research contributes to that aim by comparing three instantiations of incremental or curriculum learning. We introduce Incremental Sequence Learning, a simple incremental approach to sequence learning. Incremental Sequence Learning starts out by using only the first few steps of each sequence as training data. Each time a performance criterion has been reached, the length of the parts of the sequences used for training is increased. We introduce and make available a novel sequence learning task and data set: predicting and classifying MNIST pen stroke sequences. We find that Incremental Sequence Learning greatly speeds up sequence learning and reaches the best test performance level of regular sequence learning 20 times faster, reduces the test error by 74%, and in general performs more robustly; it displays lower variance and achieves sustained progress after all three comparison methods have stopped improving. The other instantiations of curriculum learning do not result in any noticeable improvement. A trained sequence prediction model is also used in transfer learning to the task of sequence classification, where it is found that transfer learning realizes improved classification performance compared to methods that learn to classify from scratch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑