对齐，然后记住：学习与反馈对齐的动态
Align, then memorise: the dynamics of learning with feedback alignment
摘要：
直接反馈对齐（DFA）正在成为一种有效的和生物学上合理的替代无处不在的反向传播算法用于训练深度神经网络。尽管向后传递依赖于随机反馈权重，但DFA成功地训练了最先进的模型，如Transformers。另一方面，它在训练卷积网络方面出了名的失败。理解DFA的内部运作来解释这些不同的结果仍然是难以捉摸的。在这里，我们提出了DFA的成功理论。我们首先展示了在浅网络中的学习过程分为两个步骤：对齐阶段，模型调整其权重以将近似梯度与损失函数的真实梯度对齐，然后是记忆阶段，模型专注于拟合数据。这个两步过程具有退化破坏效应：在景观中的所有低损失解决方案中，使用DFA训练的网络自然收敛到最大化梯度对齐的解决方案。我们还确定了深度线性网络中对齐的关键量：对齐矩阵的条件。后者使我们能够详细了解数据结构对对齐的影响，并对DFA在训练卷积神经网络方面的著名失败提出了一个简单的解释。在MNIST和CIFAR10上的数值实验清楚地证明了深度非线性网络中的简并破缺，并表明从网络的底层到顶层顺序地发生了搜索然后记忆的过程。
英文摘要：
Direct Feedback Alignment (DFA) is emerging as an efficient and biologically plausible alternative to the ubiquitous backpropagation algorithm for training deep neural networks. Despite relying on random feedback weights for the backward pass, DFA successfully trains state-of-the-art models such as Transformers. On the other hand, it notoriously fails to train convolutional networks. An understanding of the inner workings of DFA to explain these diverging results remains elusive. Here, we propose a theory for the success of DFA. We first show that learning in shallow networks proceeds in two steps: an alignment phase, where the model adapts its weights to align the approximate gradient with the true gradient of the loss function, is followed by a memorisation phase, where the model focuses on fitting the data. This two-step process has a degeneracy breaking effect: out of all the low-loss solutions in the landscape, a network trained with DFA naturally converges to the solution which maximises gradient alignment. We also identify a key quantity underlying alignment in deep linear networks: the conditioning of the alignment matrices. The latter enables a detailed understanding of the impact of data structure on alignment, and suggests a simple explanation for the well-known failure of DFA to train convolutional neural networks. Numerical experiments on MNIST and CIFAR10 clearly demonstrate degeneracy breaking in deep non-linear networks and show that the align-then-memorise process occurs sequentially from the bottom layers of the network to the top.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
