探索参数更新幅度对持续学习中遗忘与泛化的影响
Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning
AI总结：
本文提出了一种混合参数更新策略，通过理论分析和实验验证，改进持续学习中遗忘与泛化性能。
摘要：
参数更新的幅度被认为是持续学习中的关键因素。然而，大多数现有研究侧重于设计多样的更新策略，而对底层机制的理论理解仍然有限。因此，我们从参数更新幅度的角度 characterize 模型的遗忘，并将其正式化为由参数空间中任务特定漂移引起的知识退化，这在以前的研究中未被充分捕捉，因为这些研究假设了统一的参数空间。通过推导最小化遗忘的最优参数更新幅度，我们将两种代表性的更新范式，冻结训练和初始化训练，纳入一个受约束参数更新的优化框架中。我们的理论结果进一步揭示了在冻结训练下，参数距离较小的序列任务表现出更好的泛化能力和更少的遗忘，而不是初始化训练。这些理论见解启发了一种新的混合参数更新策略，该策略根据梯度方向自适应地调整更新幅度。在深度神经网络上的实验表明，这种混合方法优于标准训练策略，为设计高效且可扩展的持续学习算法提供了新的理论视角和实用启示。
英文摘要：
The magnitude of parameter updates are considered a key factor in continual learning. However, most existing studies focus on designing diverse update strategies, while a theoretical understanding of the underlying mechanisms remains limited. Therefore, we characterize model's forgetting from the perspective of parameter update magnitude and formalize it as knowledge degradation induced by task-specific drift in the parameter space, which has not been fully captured in previous studies due to their assumption of a unified parameter space. By deriving the optimal parameter update magnitude that minimizes forgetting, we unify two representative update paradigms, frozen training and initialized training, within an optimization framework for constrained parameter updates. Our theoretical results further reveals that sequence tasks with small parameter distances exhibit better generalization and less forgetting under frozen training rather than initialized training. These theoretical insights inspire a novel hybrid parameter update strategy that adaptively adjusts update magnitude based on gradient directions. Experiments on deep neural networks demonstrate that this hybrid approach outperforms standard training strategies, providing new theoretical perspectives and practical inspiration for designing efficient and scalable continual learning algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
