arXiv 2504.135692025-04-18cs.LG

神经网络中的贝叶斯持续学习与遗忘

Bayesian continual learning and forgetting in neural networks

Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France（巴黎-萨克雷大学，法国国家科学研究中心，纳米科学与纳米技术中心，Palaiseau）；Université Grenoble-Alpes, CEA, LETI, Grenoble, France（格勒诺布尔阿尔卑斯大学，法国原子能委员会，微电子与技术实验室，Grenoble）；Université Grenoble-Alpes, CEA, LIST, Grenoble, France（格勒诺布尔阿尔卑斯大学，法国原子能委员会，低带宽信息传输实验室，Grenoble）

Djohan Bonnet, Kellian Cottart, Tifenn Hirtzlin, Tarcisius Januel, Thomas Dalgaty, Elisa Vianello, Damien Querlioz

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对人工神经网络的灾难性遗忘与记忆问题，本文提出贝叶斯框架MESU，基于参数不确定性实现学习与遗忘的原则性结合，无需任务边界即可适应流数据，在图像分类基准中减轻遗忘并保持可塑性，性能优于现有方法。

摘要：

生物突触能轻松平衡记忆保留与灵活性，但人工神经网络仍受灾难性遗忘和灾难性记忆的极端问题困扰。本文提出突触不确定性元可塑性（MESU），这是一种根据参数不确定性更新网络参数的贝叶斯框架。该方法实现学习与遗忘的原则性结合，确保关键知识被保留，同时逐渐释放未使用或过时的信息。与可能过度受限的标准贝叶斯方法以及依赖明确任务边界的流行持续学习方法不同，MESU能无缝适应流数据。它还提供可靠的认知不确定性估计，支持分布外检测，唯一计算成本是多次采样权重以提供适当的输出统计量。图像分类基准实验表明，MESU在减轻灾难性遗忘的同时，保持对新任务的可塑性。在训练200个连续置换MNIST任务时，MESU在准确性、学习额外任务的能力和分布外数据检测方面优于现有持续学习技术。此外，由于不依赖任务边界，MESU在多种场景下持续优于传统学习技术在CIFAR-100任务的增量训练。我们的结果统一了元可塑性、贝叶斯推理和基于海森矩阵的正则化思想，为稳健的永久学习提供了受生物启发的途径。

英文摘要：

Biological synapses effortlessly balance memory retention and flexibility, yet artificial neural networks still struggle with the extremes of catastrophic forgetting and catastrophic remembering. Here, we introduce Metaplasticity from Synaptic Uncertainty (MESU), a Bayesian framework that updates network parameters according their uncertainty. This approach allows a principled combination of learning and forgetting that ensures that critical knowledge is preserved while unused or outdated information is gradually released. Unlike standard Bayesian approaches -- which risk becoming overly constrained, and popular continual-learning methods that rely on explicit task boundaries, MESU seamlessly adapts to streaming data. It further provides reliable epistemic uncertainty estimates, allowing out-of-distribution detection, the only computational cost being to sample the weights multiple times to provide proper output statistics. Experiments on image-classification benchmarks demonstrate that MESU mitigates catastrophic forgetting, while maintaining plasticity for new tasks. When training 200 sequential permuted MNIST tasks, MESU outperforms established continual learning techniques in terms of accuracy, capability to learn additional tasks, and out-of-distribution data detection. Additionally, due to its non-reliance on task boundaries, MESU outperforms conventional learning techniques on the incremental training of CIFAR-100 tasks consistently in a wide range of scenarios. Our results unify ideas from metaplasticity, Bayesian inference, and Hessian-based regularization, offering a biologically-inspired pathway to robust, perpetual learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。