基于神经极化的前向学习泛化性与稳定性改进研究
On the Improvement of Generalization and Stability of Forward-Only Learning via Neural Polarization
AI总结:
针对前向-前向算法(FFA)存在的正负样本梯度不平衡、影响模型准确率和训练稳定性的问题,本研究提出引入神经极化的Polar-FFA,实现对称梯度行为,实验证明其在准确率、收敛速度和泛化性上均优于原始FFA,且降低了超参数调优需求。
摘要:
前向学习算法作为梯度反向传播的替代方案近期受到关注,这类算法用额外的对比前向传播替代了后者的反向传播步骤。在这些方法中,所谓的Forward-Forward Algorithm(FFA,前向-前向算法)已被证明在泛化性和复杂度方面能达到具有竞争力的性能水平。使用FFA训练的网络通过对比学习,在输入真实数据(记为正样本)时最大化逐层定义的优度得分,在处理合成数据(对应负样本)时最小化该得分。然而,该算法仍存在会对模型准确率和训练稳定性产生负面影响的缺陷,这主要源于正负样本之间的梯度不平衡。为解决这一问题,本研究提出了一种新型FFA算法实现方案,命名为Polar-FFA,它通过在正负样本之间引入神经划分(极化)拓展了原始算法的公式表达。这两组神经元在接收到对应类型的数据时,都会最大化自身的优度,从而形成对称的梯度行为。为通过实验评估我们提出的Polar-FFA提升后的学习能力,我们在图像分类数据集上使用不同的激活函数和优度函数开展了多项系统性实验。结果表明,Polar-FFA在准确率和收敛速度上均优于FFA。此外,它对超参数的依赖度更低,减少了为保证最优泛化能力所需的超参数调优工作,从而能够适配更广泛的神经网络配置。
英文摘要:
Forward-only learning algorithms have recently gained attention as alternatives to gradient backpropagation, replacing the backward step of this latter solver with an additional contrastive forward pass. Among these approaches, the so-called Forward-Forward Algorithm (FFA) has been shown to achieve competitive levels of performance in terms of generalization and complexity. Networks trained using FFA learn to contrastively maximize a layer-wise defined goodness score when presented with real data (denoted as positive samples) and to minimize it when processing synthetic data (corr. negative samples). However, this algorithm still faces weaknesses that negatively affect the model accuracy and training stability, primarily due to a gradient imbalance between positive and negative samples. To overcome this issue, in this work we propose a novel implementation of the FFA algorithm, denoted as Polar-FFA, which extends the original formulation by introducing a neural division (\emph{polarization}) between positive and negative instances. Neurons in each of these groups aim to maximize their goodness when presented with their respective data type, thereby creating a symmetric gradient behavior. To empirically gauge the improved learning capabilities of our proposed Polar-FFA, we perform several systematic experiments using different activation and goodness functions over image classification datasets. Our results demonstrate that Polar-FFA outperforms FFA in terms of accuracy and convergence speed. Furthermore, its lower reliance on hyperparameters reduces the need for hyperparameter tuning to guarantee optimal generalization capabilities, thereby allowing for a broader range of neural network configurations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。