Bop及其他:二值化神经网络的二阶优化器
A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
摘要:
二进制神经网络(BNN)的优化依赖于用它们的二进制表示来近似实值权重。当前的权重更新技术使用与传统神经网络(NN)相同的方法,但需要使用符号函数的导数的近似值(因为它是Dirac-Delta函数)进行反向传播;因此,努力集中在适应全精度技术以在BNN上工作。在文献中,只有一个先前的努力解决了直接训练BNN的问题,通过使用梯度的第一个原始矩估计,并将其与阈值进行比较,以决定何时翻转权重(Bop)。在本文中,我们采取了一种类似于Adam的方法,在与阈值进行比较之前,也使用第二个原始矩估计来归一化第一个原始矩,我们称这种方法为Bop 2ndOrder。我们提出了两个版本的建议优化:有偏见的一个和一个偏差校正,每个都有自己的应用程序。此外,我们提出了一个完整的消融研究的超参数空间,以及使用的效果,他们每个人。对于这些研究,我们使用BinaryNet架构在CIFAR 10中测试了优化器。此外,我们在ImageNet 2012中使用XnorNet和BiRealNet架构测试了它的准确性。在这两个数据集中,我们的方法被证明收敛速度更快,对超参数的变化具有鲁棒性,并且获得了更好的准确度值。
英文摘要:
The optimization of Binary Neural Networks (BNNs) relies on approximating the real-valued weights with their binarized representations. Current techniques for weight-updating use the same approaches as traditional Neural Networks (NNs) with the extra requirement of using an approximation to the derivative of the sign function - as it is the Dirac-Delta function - for back-propagation; thus, efforts are focused adapting full-precision techniques to work on BNNs. In the literature, only one previous effort has tackled the problem of directly training the BNNs with bit-flips by using the first raw moment estimate of the gradients and comparing it against a threshold for deciding when to flip a weight (Bop). In this paper, we take an approach parallel to Adam which also uses the second raw moment estimate to normalize the first raw moment before doing the comparison with the threshold, we call this method Bop2ndOrder. We present two versions of the proposed optimizer: a biased one and a bias-corrected one, each with its own applications. Also, we present a complete ablation study of the hyperparameters space, as well as the effect of using schedulers on each of them. For these studies, we tested the optimizer in CIFAR10 using the BinaryNet architecture. Also, we tested it in ImageNet 2012 with the XnorNet and BiRealNet architectures for accuracy. In both datasets our approach proved to converge faster, was robust to changes of the hyperparameters, and achieved better accuracy values.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。