arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.06192cs.LGcs.AI
收藏

恰当的学习时机:在知识蒸馏中通过受生物启发的间隔效应提升泛化能力

Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation

Guanglong Sun(广龙Sun);Hongwei Yan(杨Hongwei);Liyuan Wang(王Liyuan);Qian Li(李Qian);Bo Lei(雷Bo);Yi Zhong(钟Yi)

Guanglong Sun, Hongwei Yan, Liyuan Wang, Qian Li, Bo Lei, Yi Zhong

AI总结:

受生物学习中间隔效应的启发,本研究提出Spaced KD策略,让学生模型从提前间隔训练的教师模型中蒸馏知识,通过收敛至更平坦损失曲面提升泛化性,在Tiny-ImageNet上较在线KD、自KD最高分别提升2.31%、3.34%的性能。

摘要:

知识蒸馏(KD)是训练深度神经网络(DNN)的一种强效策略。尽管其最初被提出用于从大型“教师”模型训练出更紧凑的“学生”模型,但近期许多研究聚焦于调整该策略以提升模型自身的泛化能力,例如在线KD和自KD。在此,我们提出一种易用且兼容的策略,名为Spaced KD,以同时提升在线KD和自KD的效果;在该策略中,学生模型从提前一个间隔周期训练得到的教师模型中蒸馏知识。该策略的灵感来源于生物学习与记忆中一项著名理论——间隔效应,该理论认为学习试次之间的适当间隔能显著提升学习表现。通过理论与实证分析,我们证明所提出的Spaced KD的优势源于其在随机梯度下降(SGD)过程中收敛至更平坦的损失曲面。我们开展了大量实验,验证了Spaced KD在提升DNN学习表现方面的有效性(例如,在Tiny-ImageNet数据集上,其性能相较于在线KD和自KD分别提升了最高2.31%和3.34%)。我们的代码已在GitHub发布,网址为https://github.com/SunGL001/Spaced-KD。

英文摘要:

Knowledge distillation (KD) is a powerful strategy for training deep neural networks (DNNs). Although it was originally proposed to train a more compact "student" model from a large "teacher" model, many recent efforts have focused on adapting it to promote generalization of the model itself, such as online KD and self KD. Here, we propose an accessible and compatible strategy named Spaced KD to improve the effectiveness of both online KD and self KD, in which the student model distills knowledge from a teacher model trained with a space interval ahead. This strategy is inspired by a prominent theory named spacing effect in biological learning and memory, positing that appropriate intervals between learning trials can significantly enhance learning performance. With both theoretical and empirical analyses, we demonstrate that the benefits of the proposed Spaced KD stem from convergence to a flatter loss landscape during stochastic gradient descent (SGD). We perform extensive experiments to validate the effectiveness of Spaced KD in improving the learning performance of DNNs (e.g., the performance gain is up to 2.31% and 3.34% on Tiny-ImageNet over online KD and self KD, respectively). Our codes have been released on github https://github.com/SunGL001/Spaced-KD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑