单相对比赫布学习的两个方面
Two Tales of Single-Phase Contrastive Hebbian Learning
AI总结:
本研究针对生物学合理学习算法的同步与计算开销问题,通过神经元二元组建模提出双传播算法缩小与反向传播的性能差距,并揭示其与对抗鲁棒性的关联及与稳定伴随状态方法的关系。
摘要:
寻找“生物学合理”的学习算法已聚焦于将梯度表示为活动差异的思路。然而,多数方法需高度同步(学习期间的不同阶段)并引入大量计算开销,这对其生物学合理性及神经形态计算的潜在效用提出了质疑。此外,它们通常依赖对输出单元施加微小扰动(轻推),这在噪声环境中不切实际。近期研究表明,通过将人工神经元建模为含两个反向轻推隔间的二元组,名为“双传播”的全局部学习算法可缩小与反向传播的性能差距,无需单独学习阶段或微小轻推。但该算法的缺点是数值稳定性依赖对称轻推,这在生物学和模拟实现中可能受限。本研究首先为双传播方法的目标提供坚实基础,揭示其与对抗鲁棒性的意外关联;其次证明双传播与特定伴随状态方法相关,该方法无论轻推是否对称均稳定。
英文摘要:
The search for ``biologically plausible'' learning algorithms has converged on the idea of representing gradients as activity differences. However, most approaches require a high degree of synchronization (distinct phases during learning) and introduce substantial computational overhead, which raises doubts regarding their biological plausibility as well as their potential utility for neuromorphic computing. Furthermore, they commonly rely on applying infinitesimal perturbations (nudges) to output units, which is impractical in noisy environments. Recently it has been shown that by modelling artificial neurons as dyads with two oppositely nudged compartments, it is possible for a fully local learning algorithm named ``dual propagation'' to bridge the performance gap to backpropagation, without requiring separate learning phases or infinitesimal nudging. However, the algorithm has the drawback that its numerical stability relies on symmetric nudging, which may be restrictive in biological and analog implementations. In this work we first provide a solid foundation for the objective underlying the dual propagation method, which also reveals a surprising connection with adversarial robustness. Second, we demonstrate how dual propagation is related to a particular adjoint state method, which is stable regardless of asymmetric nudging.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。