A Forward Propagation Algorithm for Online Optimization of Nonlinear Stochastic Differential Equations

Stochastic Online Linear Regression: the Forward Algorithm to Replace Ridge

We consider the problem of online linear regression in the stochastic setting. We derive high probability regret bounds for online ridge regression and the forward algorithm. This enables us to compare online regression algorithms more accurately and eliminate assumptions of bounded observations and predictions. Our study advocates for the use of the forward algorithm in lieu of ridge due to its enhanced bounds and robustness to the regularization parameter. Moreover, we explain how to integrate it in algorithms involving linear function approximation to remove a boundedness assumption without…

Formal derivation of Mesh Neural Networks with their Forward-Only gradient Propagation

This paper proposes the Mesh Neural Network (MNN), a novel architecture which allows neurons to be connected in any topology, to efficiently route information. In MNNs, information is propagated between neurons throughout a state transition function. State and error gradients are then directly computed from state updates without backward computation. The MNN architecture and the error propagation schema is formalized and derived in tensor algebra. The proposed computational model can fully supply a gradient descent process, and is potentially suitable for very large scale sparse NNs, due to it…

The Interplay Between Stability and Regret in Online Learning

This paper considers the stability of online learning algorithms and its implications for learnability (bounded regret). We introduce a novel quantity called {\em forward regret} that intuitively measures how good an online learning algorithm is if it is allowed a one-step look-ahead into the future. We show that given stability, bounded forward regret is equivalent to bounded regret. We also show that the existence of an algorithm with bounded regret implies the existence of a stable algorithm with bounded regret and bounded forward regret. The equivalence results apply to general, possibly n…

前向直接反馈对齐用于脉冲神经网络在线梯度估计

Forward Direct Feedback Alignment for Online Gradient Estimates of Spiking Neural Networks

本文提出脉冲前向直接反馈对齐（SFDFA）算法，用于在神经形态硬件上在线估计脉冲神经网络的梯度，避免反向传播，并通过比较实验证明其性能和收敛速度优于现有算法。

带前向正则化的随机神经网络增量在线学习

Incremental Online Learning of Randomized Neural Network with Forward Regularization

针对深度神经网络在线学习的非增量更新、内存增长和灾难性遗忘问题，提出随机神经网络的增量在线学习框架，引入前向正则化，实现递归权重更新，并在回归与分类任务中验证其加速学习和更低遗憾的优势。

具有自适应前向正则化的随机神经网络用于在线无任务类增量学习

Randomized Neural Network with Adaptive Forward Regularization for Online Task-free Class Incremental Learning

针对在线无任务类增量学习中的遗忘与低效问题，提出带自适应前向正则化的随机神经网络框架，通过闭式增量更新和贝叶斯自适应调参，实现抗遗忘与高性能。