arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02846cond-mat.stat-mechcond-mat.dis-nncs.LGq-bio.NC
收藏

持续学习的统计力学:变分原理与平均场势

Statistical mechanics of continual learning: variational principle and mean-field potential

Sun Yat-sen University(中山大学)

Chan Li, Zhenye Huang, Wenxuan Zou, Haiping Huang

AI总结:

针对二值神经网络多任务持续学习缺乏统一理论的问题,提出变分贝叶斯学习设定与Franz-Parisi热力学势框架,通过平均场序参数解析学习性能,并推导出优于现有元可塑性算法的考虑权重不确定性的多层网络学习算法。

摘要:

人工智能通用智能面临的一个障碍是由不同性质的多任务持续学习所设定的。近期,从机器学习和神经科学角度提出了各种启发式技巧,但它们缺乏统一的理论基础。在此,我们关注具有二值权重的单层和多层神经网络的持续学习。因此提出了一种变分贝叶斯学习设定,其中神经网络在场空间中训练,而非梯度定义不明确的离散权重空间,此外,权重不确定性被自然纳入,并在任务间调节突触资源。从物理学视角,我们将变分持续学习转化为Franz-Parisi热力学势框架,其中先前任务知识作为先验和参考。因此,我们在师生设定中将二值感知机的持续学习解释为Franz-Parisi势计算。随后可通过平均场序参数解析研究学习性能,其预测与使用随机梯度下降方法的数值实验一致。基于变分原理和隐藏层内部预激活的高斯场近似,我们还推导了考虑权重不确定性的学习算法,该算法使用多层神经网络解决二值权重的持续学习,且性能优于当前可用的元可塑性算法。我们提出的原则性框架还与弹性权重巩固、权重不确定性调节学习及受神经科学启发的元可塑性相关联,为使用深度网络的真实世界多任务学习提供了理论基础的方法。

英文摘要:

An obstacle to artificial general intelligence is set by continual learning of multiple tasks of different nature. Recently, various heuristic tricks, both from machine learning and from neuroscience angles, were proposed, but they lack a unified theory ground. Here, we focus on continual learning in single-layered and multi-layered neural networks of binary weights. A variational Bayesian learning setting is thus proposed, where the neural networks are trained in a field-space, rather than gradient-ill-defined discrete-weight space, and furthermore, weight uncertainty is naturally incorporated, and modulates synaptic resources among tasks. From a physics perspective, we translate the variational continual learning into Franz-Parisi thermodynamic potential framework, where previous task knowledge acts as a prior and a reference as well. We thus interpret the continual learning of the binary perceptron in a teacher-student setting as a Franz-Parisi potential computation. The learning performance can then be analytically studied with mean-field order parameters, whose predictions coincide with numerical experiments using stochastic gradient descent methods. Based on the variational principle and Gaussian field approximation of internal preactivations in hidden layers, we also derive the learning algorithm considering weight uncertainty, which solves the continual learning with binary weights using multi-layered neural networks, and performs better than the currently available metaplasticity algorithm. Our proposed principled frameworks also connect to elastic weight consolidation, weight-uncertainty modulated learning, and neuroscience inspired metaplasticity, providing a theory-grounded method for the real-world multi-task learning with deep networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑