CLASSP：一种通过调整抑制与稀疏性促进实现持续学习的生物启发方法
CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion
AI总结：
本文提出生物启发的持续学习方法CLASSP，通过权重调整衰减率和损失梯度阈值促进稀疏更新，在准确率与内存占用上优于EWC。
摘要：
本文介绍了一种新的生物启发训练方法，名为通过调整抑制与稀疏性促进的持续学习（CLASSP）。CLASSP基于神经科学中观察到的两个主要原理，特别是在突触传递和长时程增强（LTP）的背景下。第一个原理是权重调整上的衰减率，它被实现为AdaGrad优化算法的推广。这意味着已经接收许多更新的权重应具有较低的学习率，因为它们可能编码了关于先前所见数据的重要信息。然而，这一原理导致更新在整个模型中呈弥散分布，因为它促进了对先前未被更新的权重进行更新，而稀疏更新分布更受青睐，以便为未来任务保留未分配的权重。因此，第二个原理在损失梯度上引入了一个阈值。这通过仅当某个权重对应的损失梯度高于特定阈值时才更新该权重来促进稀疏学习，即仅更新对当前损失有显著影响的权重。这两个原理都反映了在LTP中观察到的现象，其中已观察到阈值效应和增强作用的逐渐饱和。CLASSP以Python/PyTorch类的形式实现，使其适用于任何模型。在使用计算机视觉和情感分析数据集与弹性权重巩固（EWC）进行比较时，CLASSP在准确率和内存占用方面表现出更优越的性能。
英文摘要：
This paper introduces a new biologically-inspired training method named Continual Learning through Adjustment Suppression and Sparsity Promotion (CLASSP). CLASSP is based on two main principles observed in neuroscience, particularly in the context of synaptic transmission and Long-Term Potentiation (LTP). The first principle is a decay rate over the weight adjustment, which is implemented as a generalization of the AdaGrad optimization algorithm. This means that weights that have received many updates should have lower learning rates as they likely encode important information about previously seen data. However, this principle results in a diffuse distribution of updates throughout the model, as it promotes updates for weights that haven't been previously updated, while a sparse update distribution is preferred to leave weights unassigned for future tasks. Therefore, the second principle introduces a threshold on the loss gradient. This promotes sparse learning by updating a weight only if the loss gradient with respect to that weight is above a certain threshold, i.e. only updating weights with a significant impact on the current loss. Both principles reflect phenomena observed in LTP, where a threshold effect and a gradual saturation of potentiation have been observed. CLASSP is implemented in a Python/PyTorch class, making it applicable to any model. When compared with Elastic Weight Consolidation (EWC) using Computer Vision and sentiment analysis datasets, CLASSP demonstrates superior performance in terms of accuracy and memory footprint.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
