arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.10455cs.LGcs.AIcs.CV
收藏

Learn,Unlearn and Relearn:An在线学习范式for Deep神经网络

Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks

NavInfo Europe(四维图新欧洲);Eindhoven University of Technology(埃因霍温理工大学)

Vijaya Raghavan T. Ramkumar, Elahe Arani, Bahram Zonooz

摘要:

深度神经网络(DNN)通常在提前提供完整的训练数据集的前提下进行训练。然而,在现实世界的场景中,数据通常随着时间的推移以块的形式到达。这导致了关于训练DNN的最佳策略的重要考虑,例如是否使用每个传入数据块对其进行微调(热启动),或者在新块可用时使用整个数据库从头开始重新训练它们。虽然采用后者进行训练可能是资源密集型的,但最近的工作指出了热启动模型中缺乏泛化能力。因此,为了在效率和泛化之间取得平衡,我们引入了学习,遗忘和重新学习(LURE)DNN的在线学习范例。LURE之间的unlearning阶段,选择性地忘记模型中的不需要的信息,通过权重重新初始化,在数据依赖的方式,和再学习阶段,强调学习概括的功能。我们表明,我们的训练范式在分类和Few-Shot设置中提供了跨数据集的一致性能增益。我们进一步表明,它会导致更强大和良好的校准模型。

英文摘要:

Deep neural networks (DNNs) are often trained on the premise that the complete training data set is provided ahead of time. However, in real-world scenarios, data often arrive in chunks over time. This leads to important considerations about the optimal strategy for training DNNs, such as whether to fine-tune them with each chunk of incoming data (warm-start) or to retrain them from scratch with the entire corpus of data whenever a new chunk is available. While employing the latter for training can be resource-intensive, recent work has pointed out the lack of generalization in warm-start models. Therefore, to strike a balance between efficiency and generalization, we introduce Learn, Unlearn, and Relearn (LURE) an online learning paradigm for DNNs. LURE interchanges between the unlearning phase, which selectively forgets the undesirable information in the model through weight reinitialization in a data-dependent manner, and the relearning phase, which emphasizes learning on generalizable features. We show that our training paradigm provides consistent performance gains across datasets in both classification and few-shot settings. We further show that it leads to more robust and well-calibrated models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑