CBP：使用伪拉格朗日乘子方法的具有权重精度约束的反向传播
CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
摘要：
误差的反向传播（反向传播）是一种最小化目标函数（例如，损失函数），通过识别权重和偏置的最佳集合。通常需要对重量精度施加约束以减轻硬件上的禁止性工作负载。尽管反向传播的显着成功，算法本身是不能够考虑这样的约束，除非同时应用额外的算法。为了解决这个问题，我们提出了约束反向传播（CBP）算法的基础上的伪拉格朗日乘子法，以获得最佳的一组权重，满足给定的一组约束。所提出的CBP算法的定义特征是利用拉格朗日函数（损失函数加约束函数）作为其目标函数。我们考虑了各种类型的约束--二进制、三进制、一位移位和两位移位权重约束。作为后训练方法，CBP应用于ImageNet上的AlexNet，ResNet-18，ResNet-50和GoogLeNet，这些都是使用传统的反向传播进行预训练的。对于所有情况，所提出的算法都优于ImageNet上的最先进方法，例如，ResNet-18、ResNet-50和GoogLeNet在二进制权重下的top-1准确率分别为66.6%、74.4%和64.0%。这突出了CBP作为一种学习算法，通过采用适当的约束函数，以最小的性能损失来解决各种约束。
英文摘要：
Backward propagation of errors (backpropagation) is a method to minimize objective functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints on weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is not capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on a pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) as its objective function. We considered various types of constraints--binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post-training method, CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet, which were pre-trained using the conventional backpropagation. For all cases, the proposed algorithm outperforms the state-of-the-art methods on ImageNet, e.g., 66.6%, 74.4%, and 64.0% top-1 accuracy for ResNet-18, ResNet-50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm to address diverse constraints with the minimal performance loss by employing appropriate constraint functions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
