AI 中文总结
研究神经网络可塑性度量，引入基于通用压缩理论的局部冗余，将其定义为局部模型族最坏情况冗余，证明期望平方梯度范数可作下界，实验表明该度量比现有方法更能预测下游性能并助于预训练检查点选择。
AI 中文摘要
可塑性——神经网络适应新任务的能力——对持续学习和迁移学习至关重要。现有度量，如实效秩、死亡神经元比例和权重范数，缺乏理论基础，与新任务性能的相关性较差。我们引入局部冗余，一种源自通用压缩理论的信息论度量。我们将局部冗余定义为局部模型族（沿梯度方向的无穷小邻域中的参数）的最坏情况冗余，并表明这是一种可塑性的原则性度量。虽然局部冗余难以精确计算，但我们证明在合成记忆任务上的期望平方梯度范数提供了一个可有效计算的下界。在持续图像分类和时间序列迁移学习上的实验表明，局部冗余比现有度量能更好地预测下游性能，并能在验证损失平稳时进行预训练检查点选择。
英文摘要
Plasticity -- a neural network's ability to adapt to new tasks -- is critical for continual and transfer learning. Existing measures, such as effective rank, dead neuron fraction, and weight norm, lack theoretical grounding and correlate poorly with performance on new tasks. We introduce local redundancy, an information-theoretic measure derived from universal compression theory. We define local redundancy as the worst-case redundancy of a local model family -- parameters in an infinitesimal neighborhood along gradient directions -- and show this is a principled measure of plasticity. Although local redundancy is intractable to compute exactly, we prove that the expected squared gradient norm on a synthetic memorization task provides an efficiently computable lower bound. Experiments on continual image classification and time series transfer learning demonstrate that local redundancy predicts downstream performance better than existing measures and enables pretraining checkpoint selection where validation loss plateaus.
Comments13 pages, 7 figures. ICML 2026 (Spotlight)
