arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.09708cs.CVcs.AIcs.LG
收藏

DenseShift:迈向准确且高效的低比特二次幂量化

DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization

Noah’s Ark Lab, Huawei Technologies(华为技术有限公司诺亚方舟实验室);University of Montreal(蒙特利尔大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所)

Xinlin Li, Bang Liu, Rui Heng Yang, Vanessa Courville, Chao Xing, Vahid Partovi Nia

AI总结:

提出DenseShift网络,通过无零位移机制、符号-尺度分解和低方差随机初始化策略,解决低比特二次幂量化网络的精度与推理效率问题,在视觉和语音任务中实现与全精度网络相当的性能且具备强迁移能力。

摘要:

由于深度神经网络对资源的需求不断增加,将其高效部署在低资源边缘设备上具有挑战性。为解决此问题,研究人员提出了无乘法神经网络,如Power-of-Two quantization(二次幂量化,也称为Shift网络),旨在减少内存使用并简化计算。然而,现有的低比特Shift网络准确度不及全精度网络,通常受限于权重范围编码方案和量化损失。本文提出DenseShift网络,显著提升了Shift网络的准确度,在视觉和语音应用中实现了与全精度网络相当的性能。此外,我们引入一种使用非量化浮点激活来部署高效DenseShift网络的方法,相比现有方法获得了1.6倍加速。为此,我们证明低比特Shift网络中的零权重值对模型容量无贡献,且对推理计算产生负面影响。为解决此问题,我们提出zero-free shifting mechanism(无零位移机制),简化推理并增加模型容量。我们进一步提出sign-scale decomposition(符号-尺度分解)设计以增强训练效率,以及low-variance random initialization(低方差随机初始化)策略以提升模型迁移学习性能。我们在各种计算机视觉和语音任务上的大量实验表明,DenseShift优于现有的低比特无乘法网络,并实现了与全精度网络相当的性能。此外,我们提出的方法展现出强大的迁移学习性能且准确率无下降。代码已在GitHub发布。

英文摘要:

Efficiently deploying deep neural networks on low-resource edge devices is challenging due to their ever-increasing resource requirements. To address this issue, researchers have proposed multiplication-free neural networks, such as Power-of-Two quantization, or also known as Shift networks, which aim to reduce memory usage and simplify computation. However, existing low-bit Shift networks are not as accurate as their full-precision counterparts, typically suffering from limited weight range encoding schemes and quantization loss. In this paper, we propose the DenseShift network, which significantly improves the accuracy of Shift networks, achieving competitive performance to full-precision networks for vision and speech applications. In addition, we introduce a method to deploy an efficient DenseShift network using non-quantized floating-point activations, while obtaining 1.6X speed-up over existing methods. To achieve this, we demonstrate that zero-weight values in low-bit Shift networks do not contribute to model capacity and negatively impact inference computation. To address this issue, we propose a zero-free shifting mechanism that simplifies inference and increases model capacity. We further propose a sign-scale decomposition design to enhance training efficiency and a low-variance random initialization strategy to improve the model's transfer learning performance. Our extensive experiments on various computer vision and speech tasks demonstrate that DenseShift outperforms existing low-bit multiplication-free networks and achieves competitive performance compared to full-precision networks. Furthermore, our proposed approach exhibits strong transfer learning performance without a drop in accuracy. Our code was released on GitHub.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑