理解不同模型容量下持续学习中复习规模的作用
Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacities
AI总结：
本研究探讨了不同模型容量下复习规模对持续学习的影响，揭示了复习可能损害模型适应性及记忆保留的非直观结论。
摘要：
复习是缓解灾难性遗忘的关键技术之一，因其简单和实用性而被广泛应用于持续学习算法中。然而，复习规模如何影响学习动态的理论理解仍然有限。为解决这一差距，我们将基于复习的持续学习公式化为一个多维有效性驱动的迭代优化问题，为各种性能指标提供统一的表征。在此框架内，我们从复习规模的角度推导了适应性、可记忆性和泛化性的闭式分析。我们的结果揭示了几个引人注目且反直觉的发现。首先，复习可能会损害模型的适应性，这与传统上认为的益处形成鲜明对比。其次，增加复习规模并不一定提高记忆保留。当任务相似且噪声水平较低时，记忆误差表现出一个逐渐减小的下限。最后，我们通过数值模拟和在多个现实世界数据集上的扩展分析验证了这些见解，揭示了持续学习中复习机制的统计模式。
英文摘要：
Rehearsal is one of the key techniques for mitigating catastrophic forgetting and has been widely adopted in continual learning algorithms due to its simplicity and practicality. However, the theoretical understanding of how rehearsal scale influences learning dynamics remains limited. To address this gap, we formulate rehearsal-based continual learning as a multidimensional effectiveness-driven iterative optimization problem, providing a unified characterization across diverse performance metrics. Within this framework, we derive a closed-form analysis of adaptability, memorability, and generalization from the perspective of rehearsal scale. Our results uncover several intriguing and counterintuitive findings. First, rehearsal can impair model's adaptability, in sharp contrast to its traditionally recognized benefits. Second, increasing the rehearsal scale does not necessarily improve memory retention. When tasks are similar and noise levels are low, the memory error exhibits a diminishing lower bound. Finally, we validate these insights through numerical simulations and extended analyses on deep neural networks across multiple real-world datasets, revealing statistical patterns of rehearsal mechanisms in continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
