arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1706.04499cs.LGstat.ML
收藏

SEARNN:训练具有全局-局部损失的RNN

SEARNN: Training RNNs with Global-Local Losses

Département d’informatique de l’ENS(巴黎高等师范学院计算机系);INRIA(法国国家信息与自动化研究所);École normale supérieure(高等师范学院);CNRS(法国国家科学研究中心);PSL Research University(巴黎文理研究大学);National Research University Higher School of Economics(俄罗斯国立高等经济大学);Université de Montréal(蒙特利尔大学);Montreal Institute for Learning Algorithms (MILA)(蒙特利尔学习算法研究所);Canadian Institute for Advanced Research (CIFAR)(加拿大高级研究所)

Rémi Leblond, Jean-Baptiste Alayrac, Anton Osokin, Simon Lacoste-Julien

摘要:

我们提出了SEARNN,一种新的训练算法的递归神经网络(RNNs)的启发“学习搜索”(L2 S)的方法来结构化预测。RNN在诸如机器翻译或解析的结构化预测应用中已经取得了广泛的成功,并且通常使用最大似然估计(MLE)进行训练。不幸的是,这种训练损失并不总是测试错误的适当替代品:通过仅最大化地面真值概率,它未能利用结构化损失提供的丰富信息。此外,它还引入了训练和预测之间的差异(例如暴露偏差),这可能会损害测试性能。相反,SEARNN利用类似测试的搜索空间探索来引入更接近测试误差的全局-局部损失。我们首先展示了在两个不同的任务:OCR和拼写校正MLE的性能提高。然后,我们提出了一个子采样策略,使SEARNN扩展到大的词汇量。这使我们能够验证我们的方法在机器翻译任务上的好处。

英文摘要:

We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the "learning to search" (L2S) approach to structured prediction. RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE). Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses. Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance. Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error. We first demonstrate improved performance over MLE on two different tasks: OCR and spelling correction. Then, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes. This allows us to validate the benefits of our approach on a machine translation task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑