论持续学习中模型容量的动态特性
On Understanding of the Dynamics of Model Capacity in Continual Learning
AI总结：
本文引入CLEMC刻画持续学习中稳定性-可塑性平衡的动态行为，通过差分方程建模证明有效容量本质非平稳，并在多种架构上验证了任务分布差异导致模型表示新任务能力下降。
摘要：
稳定性-可塑性困境与神经网络（NN）的容量（即其表示任务的能力）密切相关，是持续学习（CL）中的基本挑战。在此背景下，我们引入了CL的有效模型容量（CLEMC），用以刻画稳定性-可塑性平衡点的动态行为。我们构建了一个差分方程来建模NN、任务数据和优化过程之间相互作用的演化。随后，我们利用CLEMC证明，有效容量及其稳定性-可塑性平衡点本质上是非平稳的。我们表明，无论NN架构或优化方法如何，当新任务分布与先前任务不同时，NN表示新任务的能力都会下降。我们进行了广泛实验以支持理论发现，涵盖多种架构：从小型前馈网络和卷积网络，到中等规模的图神经网络，再到拥有数百万参数的基于Transformer的大型语言模型。
英文摘要：
The stability-plasticity dilemma, closely related to a neural network's (NN) capacity-its ability to represent tasks-is a fundamental challenge in continual learning (CL). Within this context, we introduce CL's effective model capacity (CLEMC) that characterizes the dynamic behavior of the stability-plasticity balance point. We develop a difference equation to model the evolution of the interplay between the NN, task data, and optimization procedure. We then leverage CLEMC to demonstrate that the effective capacity-and, by extension, the stability-plasticity balance point is inherently non-stationary. We show that regardless of the NN architecture or optimization method, a NN's ability to represent new tasks diminishes when incoming task distributions differ from previous ones. We conduct extensive experiments to support our theoretical findings, spanning a range of architectures-from small feedforward network and convolutional networks to medium-sized graph neural networks and transformer-based large language models with millions of parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
