arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1212.0901cs.LG
收藏

递归网络优化研究进展

Advances in Optimizing Recurrent Networks

U. Montreal(蒙特利尔大学)

Yoshua Bengio, Nicolas Boulanger-Lewandowski, Razvan Pascanu

摘要:

经过十多年的时间,在递归神经网络领域的研究活动相对较少,这里将回顾几个新的发展,这些发展在理解和技术解决方案方面都取得了实质性进展,以实现更有效的递归网络训练。这些进步受到了围绕深度学习的优化问题的推动,并与之相关。虽然递归网络在原则上可以表示建模序列方面非常强大,但它们的训练受到关于长期依赖性学习的同一问题的两个方面的困扰。这里报告的实验评估使用裁剪梯度,跨越更长的时间范围与漏积分,先进的动量技术,使用更强大的输出概率模型,并鼓励稀疏梯度,以帮助对称性破缺和信用分配。实验是在文本和音乐数据上进行的,并展示了这些技术在普遍改善训练和测试错误方面的综合效果。

英文摘要:

After a more than decade-long period of relatively little research activity in the area of recurrent neural networks, several new developments will be reviewed here that have allowed substantial progress both in understanding and in technical solutions towards more efficient training of recurrent networks. These advances have been motivated by and related to the optimization issues surrounding deep learning. Although recurrent networks are extremely powerful in what they can in principle represent in terms of modelling sequences,their training is plagued by two aspects of the same issue regarding the learning of long-term dependencies. Experiments reported here evaluate the use of clipping gradients, spanning longer time ranges with leaky integration, advanced momentum techniques, using more powerful output probability models, and encouraging sparser gradients to help symmetry breaking and credit assignment. The experiments are performed on text and music data and show off the combined effects of these techniques in generally improving both training and test error.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑