通过活动和误差几何实现条件直接反馈对齐
Conditioned Direct Feedback Alignment via Activity and Error Geometry
- Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University（哈佛大学坎普纳自然与人工智能研究所）
- Howard Hughes Medical Institute, Harvard Medical School（哈佛医学院霍华德·休斯医学研究所）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
研究直接反馈对齐（DFA）训练失败模式，通过分析各向异性进入局部权重更新的方式，提出条件DFA，经实验验证其在不同模型上有效，还介绍了归一化DFA家族及相关计算动机，强调是对局部外积规则失败的因子级研究。
AI中文摘要：
直接反馈对齐（DFA）通过输出误差的固定随机投影来训练隐藏层，避免了反向传播（BP）的转置权重反向传播。我们研究了DFA训练中一种与反馈质量不同的失败模式：局部权重更新由外积计算，因此各向异性可以通过其突触前活动因子或局部误差因子进入。我们在受控合成机制下的分析分离出了第一种失败模式，并表明当高方差方向包含与任务无关的干扰时，活动条件增益约为40个百分点。三个清晰的验证分离出了不同的机制：误差条件使原始DFA提高了1.77 - 7.53个百分点，并且组合独立选择的活动和误差因子比活动条件增加了0.40 - 0.90个百分点。这些结果在tanh/一对多MNIST和预注册的Fashion-MNIST上成立，并在ReLU/softmax MNIST模型的八个新种子上重复。这种分解产生了一个对称的块局部归一化DFA（nDFA）家族：活动nDFA通过逆活动二阶矩进行右预条件，误差nDFA通过逆局部误差二阶矩进行左预条件，而K-nDFA应用两个因子并分别调整阻尼。线性化的对齐后计算给出了精确的输入侧谱恒等式和双边规则的克罗内克因子动机，而范数匹配排除了标量步长解释。当欠阻尼时，误差因子很脆弱，BatchNorm是活动侧的强大替代方案，卷积网络的增益仍然是部分的。因此，我们将条件DFA构建为对局部外积规则何时失败的因子级研究，而不是作为BP的一般替代方案或全层卷积信用分配的解决方案。
英文摘要：
Direct feedback alignment (DFA) trains hidden layers with fixed random projections of the output error. Even when this feedback provides useful credit, unequal scales across activity or error directions can distort the local update. We study conditioned DFA (nDFA), a family that adapts established inverse-moment preconditioning to either side of this update. An aligned linear analysis describes how activity conditioning changes spectral learning rates and early-stopping risk. Synthetic experiments show gains from both stabilization and changes in update direction. Confirmatory CIFAR-10 experiments show that activity conditioning, including its FOOF formulation, improves tuned DFA at matched measured training time. Controls support a role for centered correlations beyond the tested mean and diagonal alternatives, while early-only conditioning retains most of the benefit with less training work. Conditioning also benefits backpropagation, and timing interventions do not establish a mechanism specific to random feedback. Error conditioning improves short-budget models and changes update directions, but its small additional improvement under stable training does not survive correction for multiple comparisons. Predicted benefits on image-background benchmarks are not confirmed. These findings establish practical benefits and important limits of conditioning learning with fixed random feedback.
