arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.01435cs.CV
收藏

具有自适应位宽的深度网络量化改进技术

Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths

Boston University(波士顿大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究);MIT(麻省理工学院)

Ximeng Sun, Rameswar Panda, Chun-Fu Chen, Naigang Wang, Bowen Pan, Kailash Gopalakrishnan, Aude Oliva, Rogerio Feris, Kate Saenko

摘要:

具有自适应位宽的量化深度网络是一种很有前途的技术,可以跨许多设备和资源约束进行有效推理。与重复量化过程并针对不同约束训练不同模型的静态方法相比,自适应量化使我们能够在推理过程中灵活调整单个深度网络的位宽,以便在不同场景中即时适应。虽然现有的研究表明,令人鼓舞的结果,共同的图像分类基准,本文研究如何更有效地训练这样的自适应网络。具体来说,我们提出了两种新的技术,用于量化深度神经网络与自适应位宽的权重和激活。首先,我们提出了一个协作策略,选择一个高精度的教师将知识传递给低精度的学生,同时联合优化所有位宽的模型。其次,为了有效地传递知识,我们开发了一种动态块交换方法,将精度较低的学生网络中的块随机替换为精度较高的教师网络中的相应块。多个图像分类数据集,包括视频分类基准的第一次,广泛的实验,以及证明了我们的方法比国家的最先进的方法的有效性。

英文摘要:

Quantizing deep networks with adaptive bit-widths is a promising technique for efficient inference across many devices and resource constraints. In contrast to static methods that repeat the quantization process and train different models for different constraints, adaptive quantization enables us to flexibly adjust the bit-widths of a single deep network during inference for instant adaptation in different scenarios. While existing research shows encouraging results on common image classification benchmarks, this paper investigates how to train such adaptive networks more effectively. Specifically, we present two novel techniques for quantizing deep neural networks with adaptive bit-widths of weights and activations. First, we propose a collaborative strategy to choose a high-precision teacher for transferring knowledge to the low-precision student while jointly optimizing the model with all bit-widths. Second, to effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network. Extensive experiments on multiple image classification datasets including video classification benchmarks for the first time, well demonstrate the efficacy of our approach over state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑