发表机构
Saitama University（埼玉大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出基于逆激活回归的神经元合并方法，结合聚类与最小二乘估计，实现sigmoid网络训练后压缩，并验证权重与激活信息在合并中的不同作用。
AI 中文摘要
随着神经网络规模的不断扩大，模型压缩在有限计算资源下实现高效推理变得越来越重要。结构化剪枝方法会移除那些被估计为不太重要的神经元或通道，但被移除的单元可能仍然包含有用的信息。从对训练好的网络进行粗粒化的角度来看，当多个神经元自由度被合并时，值得探究应该保留哪些信息。本文讨论了基于聚类的合并方法，用于压缩训练好的神经网络。除了无需数据的贡献加权平均方法外，我们还提出了神经元合并方法，其中通过逆激活函数将神经元响应映射回预激活空间，并使用最小二乘法估计每个代表性神经元的权重和偏置。我们还考察了使用实际训练输入的数据辅助策略和使用随机生成输入的无数据策略。比较结果提供了经验证据，在测试的sigmoid网络中，权重信息对聚类特别有用，而激活信息在合并过程中对代表性神经元重建有用。
英文摘要
As neural networks continue to grow in scale, model compression is becoming increasingly important for efficient inference under limited computational resources. Structured pruning methods remove neurons or channels that are estimated to be less important, but the removed units may still contain useful information. From the viewpoint of coarse-graining a trained network, it is valuable to ask which information should be retained when multiple neuronal degrees of freedom are consolidated. In this paper, we discuss cluster-based merging methods for compression of trained neural networks. In addition to a data-free contribution-weighted averaging method, we propose neuron-merging methods in which neuron responses are mapped back to the pre-activation space via the inverse activation function, and the weights and biases of each representative neuron are estimated using the least-squares method. We also examine both a data-assisted strategy with actual training inputs and a data-free strategy using randomly generated inputs. The comparisons provide empirical evidence, in the tested sigmoid networks, that weight information is particularly useful for clustering whereas activation information is useful for representative-neuron reconstruction in the merging process.
Comments9 pages, 7 figures
