arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1703.03129cs.LG
收藏

学会记住罕见事件

Learning to Remember Rare Events

Google Brain(谷歌大脑);Georgia Tech(佐治亚理工学院)

Łukasz Kaiser, Ofir Nachum, Aurko Roy, Samy Bengio

摘要:

尽管最近取得了进展,但记忆增强的深度神经网络在终身和一次性学习方面仍然有限,特别是在记住罕见事件方面。我们提出了一种用于深度学习的大规模终身记忆模块。该模块利用快速最近邻算法来提高效率,从而扩展到大内存大小。除了最近邻查询,该模块是完全可区分的,并且是端到端训练的,没有额外的监督。它以终身的方式运作,即,而不需要在训练期间重置它。我们的记忆模块可以很容易地添加到监督神经网络的任何部分。为了展示其多功能性,我们将其添加到许多网络中,从在图像分类上测试的简单卷积网络到深度序列到序列和递归卷积模型。在所有情况下,增强的网络都获得了记忆和终身一次性学习的能力。我们的模块记住了过去显示的数千个步骤的训练样本,并且可以成功地从它们中进行归纳。我们在Omniglot数据集上为一次性学习设置了新的最先进技术,并首次在大规模机器翻译任务中展示了循环神经网络中的终身一次性学习。

英文摘要:

Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑