发表机构
Athena Research Center; National Technical University of Athens（雅典娜研究中心; 雅典国家技术大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对形态神经网络权重雅可比稀疏、参数梯度差的问题，提出受MWU启发的关联训练方法，在9个基准测试中8个获改进，最高提升32.84个百分点且降低运行变异性。
AI 中文摘要
神经网络通常使用一阶方法和反向传播进行训练。对于权重雅可比矩阵稀疏、参数梯度可能较差的形态层，这种方法是否最优尚不明确。本研究受乘法权重更新（MWU）方案启发，提出了一种针对形态神经网络的新型权重更新方法。我们将每个形态感知机视为对数空间中专家建议学习问题的实例，采用基于关联的奖励机制，该机制会偏向与期望输出变化对齐的输入，无论强梯度信号是否已传递至其权重。我们通过将全连接层作为独立模型或更大Transformer网络的组成部分进行训练，对所提方法进行了实证评估。在9个基准测试中，关联训练在8个基准上取得了改进，提升幅度最高达32.84个百分点，同时大幅降低了运行间的变异性。
英文摘要
Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme. We view each morphological perceptron as an instance of the learning from experts' advice problem in logarithmic space, and use a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight. We empirically evaluate our approach by training fully connected layers both as stand-alone models and as parts of larger transformer networks. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability.
CommentsPreprint
