arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.00155cs.LGcs.AIcs.NE
收藏

联结主义网络中的偶然遗忘

Fortuitous Forgetting in Connectionist Networks

Mila(米拉);Université de Montréal(蒙特利尔大学);Google Research, Brain Team(谷歌研究院大脑团队)

Hattie Zhou, Ankit Vani, Hugo Larochelle, Aaron Courville

摘要:

遗忘通常被视为人类和机器学习中不需要的特性。然而,我们认为遗忘实际上对学习是有利的。我们引入“遗忘和重新学习”作为一个强大的范例,塑造人工神经网络的学习轨迹。在这个过程中,遗忘步骤选择性地从模型中删除不需要的信息,重新学习步骤加强在不同条件下始终有用的特征。遗忘-再学习框架统一了图像分类和语言涌现文献中的许多现有迭代训练算法,并允许我们理解这些算法在不成比例地遗忘不期望的信息方面的成功。我们利用这种理解,通过设计更有针对性的遗忘操作来改进现有算法。从我们的分析中获得的见解为神经网络中迭代训练的动态提供了一个连贯的观点,并为性能改进提供了一条清晰的道路。

英文摘要:

Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑