基于贝叶斯神经网络中相关性参数更新的高效持续学习修正正则化
Revised Regularization for Efficient Continual Learning through Correlation-Based Parameter Update in Bayesian Neural Networks
AI总结：
本文提出一种基于贝叶斯神经网络和变分推断的持续学习方法，通过参数分布学习降低存储、引入修正正则化项及重要性加权证据下界来缓解灾难性遗忘，并在多种数据集上验证了其优越性能。
摘要：
我们提出了一种基于贝叶斯神经网络并使用变分推断的持续学习算法，旨在克服现有方法的若干缺陷。具体而言，在持续学习场景中，为了保留知识而在每一步存储网络参数会带来挑战。由于对过去数据集的访问有限，缓解灾难性遗忘的迫切需求进一步加剧了这一困难，使得在所有会话中维持网络参数与数据集之间的对应关系变得复杂。当前使用KL散度的变分推断方法在不确定节点更新和某些节点的耦合扰动期间存在灾难性遗忘的风险。为了解决这些挑战，我们提出以下策略。为减少全连接层参数的存储，我们提出了一种参数分布学习方法，可显著降低存储需求。在采用变分推断的持续学习框架中，本研究引入了一个专门针对参数均值和方差的动态及总体的正则化项。该项旨在保留KL散度的优势，同时解决相关挑战。为确保网络参数与数据之间的正确对应，我们的方法引入了一个重要性加权的证据下界项，以捕获数据与参数之间的相关性。这使得能够存储公共和独特的参数超空间基。所提出的方法将参数空间划分为公共子空间和独特子空间，并给出有效的后向和前向知识迁移条件，从而阐明网络参数与数据集之间的对应关系。实验结果证明了我们的方法在多种数据集以及各种顺序数据集组合上的有效性，与现有方法相比取得了更优越的性能。
英文摘要：
We propose a Bayesian neural network-based continual learning algorithm using Variational Inference, aiming to overcome several drawbacks of existing methods. Specifically, in continual learning scenarios, storing network parameters at each step to retain knowledge poses challenges. This is compounded by the crucial need to mitigate catastrophic forgetting, particularly given the limited access to past datasets, which complicates maintaining correspondence between network parameters and datasets across all sessions. Current methods using Variational Inference with KL divergence risk catastrophic forgetting during uncertain node updates and coupled disruptions in certain nodes. To address these challenges, we propose the following strategies. To reduce the storage of the dense layer parameters, we propose a parameter distribution learning method that significantly reduces the storage requirements. In the continual learning framework employing variational inference, our study introduces a regularization term that specifically targets the dynamics and population of the mean and variance of the parameters. This term aims to retain the benefits of KL divergence while addressing related challenges. To ensure proper correspondence between network parameters and the data, our method introduces an importance-weighted Evidence Lower Bound term to capture data and parameter correlations. This enables storage of common and distinctive parameter hyperspace bases. The proposed method partitions the parameter space into common and distinctive subspaces, with conditions for effective backward and forward knowledge transfer, elucidating the network-parameter dataset correspondence. The experimental results demonstrate the effectiveness of our method across diverse datasets and various combinations of sequential datasets, yielding superior performance compared to existing approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
