迈向更深层预测编码神经网络的训练
Towards the Training of Deeper Predictive Coding Neural Networks
AI总结：
本文针对预测编码网络在深层架构中的性能退化问题，提出精度加权优化、权重更新机制和辅助神经元三种方法，使深度模型性能与反向传播相当。
摘要：
预测编码网络是一种通过迭代能量最小化过程进行推理的神经模型，其操作在空间和时间上都是局部的。虽然在浅层架构中有效，但在超过五到七层时，它们会遭受显著的性能退化。在这项工作中，我们表明这种退化是由权重更新期间层间指数级不平衡的误差，以及来自前几层的预测在指导更深层更新方面效果不佳所导致的。此外，当使用跳跃连接训练模型时，残差传播的能量比主路径传播的能量更快地到达更高层，从而影响测试准确率。我们通过引入一种新颖的精度加权潜变量优化来解决第一个问题，该优化在松弛阶段平衡误差分布；通过提出一种新颖的权重更新机制来减少更深层中的误差累积，从而解决第二个问题；并通过使用辅助神经元来减缓残差连接中能量的传播，从而解决第三个问题。实验上，我们的方法在深度模型（如ResNet）上实现了与反向传播相当的性能，为预测编码在复杂任务中的应用开辟了新的可能性。
英文摘要：
Predictive coding networks are neural models that perform inference through an iterative energy minimization process, whose operations are local in space and time. While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers. In this work, we show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers. Furthermore, when training models with skip connections, the energy propagated by the residuals reaches higher layers faster than that propagated by the main pathway, affecting test accuracy. We address the first issue by introducing a novel precision-weighted optimization of latent variables that balances error distributions during the relaxation phase, the second issue by proposing a novel weight update mechanism that reduces error accumulation in deeper layers, and the third one by using auxiliary neurons that slow down the propagation of the energy in the residual connections. Empirically, our methods achieve performance comparable to backpropagation on deep models such as ResNets, opening new possibilities for predictive coding in complex tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
