AI 中文总结
本文提出一种训练后早停机制，通过预测二值神经网络累积量的最终符号，可减少VGG11在CIFAR-10上的运算量，仅造成小幅准确率下降。
AI 中文摘要
二值神经网络在受限部署场景中极具吸引力，可实现小内存占用与低功耗推理。对于二值激活，点积变为符号控制的加法或减法，但运算数量不变；实际上，每个神经元或输出通道仍会累积所有输入，即便仅保留符号，这常造成资源浪费。随着累积过程推进，运行中的部分和常大幅偏离零点，使得最终符号在远未处理完最后一项时就已高度可预测；此后评估的每个贡献项会改变和的数值，但不改变最终输出激活。本文将该观察转化为一种训练后早停机制：我们刻画训练数据集上运行累积量的行为，并利用该信息尽早预测最终符号，无需重新训练任何模型参数。我们在权重的理想排序下统计运算数量：对应用于CIFAR-10数据集的VGG11模型，该方法可移除最深卷积层86.6%的累积项，仅造成0.37个点的准确率下降；若同时应用于最深的三个卷积层，可减少全网络25%的算术运算，仅造成1.36个点的准确率下降。
英文摘要
Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference. For binary activations, the dot products become sign-controlled additions or subtractions, but the number of operations is unchanged. Indeed, every neuron or output channel still accumulates all of its input, even though only the sign will be retained, which is often wasteful. As the accumulation progresses, the running partial sum frequently drifts so far from zero that its final sign becomes highly predictable long before the last term is reached; every contribution evaluated after that point changes the value of the sum but not the final output activation. This paper turns this observation into a post-training early-stopping mechanism. We characterize the behavior of the running accumulations on the training dataset and use this information to predict the final sign as soon as possible. No model parameter is retrained. We count the number of operations under an idealized ordering of weights. On VGG11 applied to the CIFAR-10 dataset, the method removes $86.6\%$ of the accumulation terms of the deepest convolution for a $0.37$-point accuracy drop, and $25\%$ of the full-network arithmetic when used on the three deepest convolutions simultaneously, for a $1.36$-point drop.
Comments9 pages, 3 figures, 1 table
