arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.04236cs.LGcs.AI
收藏

ECQ$^{\ ext{x}}$:面向低比特与稀疏深度神经网络的可解释性驱动量化

ECQ$^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs

Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫兹研究所);BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD——柏林学习与数据基础研究所)

Daniel Becking, Maximilian Dreyer, Wojciech Samek, Karsten Müller, Sebastian Lapuschkin

AI总结:

本文提出ECQ$^{\ ext{x}}$量化方法,结合可解释AI的逐层相关性传播与信息论熵优化,在2-5比特超低精度下生成稀疏DNN,实现最高103倍压缩且保持或提升模型性能。

摘要:

深度神经网络(DNNs)在各种应用中取得了显著成功,但随之而来的是网络参数和算术运算量的大幅增加。这种内存和计算需求的增长使得深度学习在移动设备等资源受限的硬件平台上难以部署。近期研究致力于在尽可能保持模型性能的同时降低这些开销,相关技术包括参数缩减技术、参数量化和无损压缩技术。在本章中,我们提出并描述了一种新颖的DNN量化范式:我们的方法利用了可解释人工智能(XAI)的概念和信息论的概念:权重分配函数不再仅仅基于权重到量化簇的距离来赋值,而是额外考虑了通过逐层相关性传播(LRP)获得的权重相关性以及簇的信息内容(熵优化)。最终目标是在信息内容最高的量化簇中保留最相关的权重。实验结果表明,这种新颖的熵约束与XAI调整量化(ECQ$^{\ ext{x}}$)方法能够生成超低精度(2-5比特)且同时稀疏的神经网络,同时保持甚至提升模型性能。由于参数精度降低和大量零元素的存在,生成的网络在文件大小方面具有高度可压缩性,与全精度未量化DNN模型相比压缩率最高可达$103\ imes$。我们的方法在不同类型的模型和数据集上进行了评估(包括Google Speech Commands、CIFAR-10和Pascal VOC),并与先前工作进行了比较。

英文摘要:

The remarkable success of deep neural networks (DNNs) in various applications is accompanied by a significant increase in network parameters and arithmetic operations. Such increases in memory and computational demands make deep learning prohibitive for resource-constrained hardware platforms such as mobile devices. Recent efforts aim to reduce these overheads, while preserving model performance as much as possible, and include parameter reduction techniques, parameter quantization, and lossless compression techniques. In this chapter, we develop and describe a novel quantization paradigm for DNNs: Our method leverages concepts of explainable AI (XAI) and concepts of information theory: Instead of assigning weight values based on their distances to the quantization clusters, the assignment function additionally considers weight relevances obtained from Layer-wise Relevance Propagation (LRP) and the information content of the clusters (entropy optimization). The ultimate goal is to preserve the most relevant weights in quantization clusters of highest information content. Experimental results show that this novel Entropy-Constrained and XAI-adjusted Quantization (ECQ$^{\text{x}}$) method generates ultra low-precision (2-5 bit) and simultaneously sparse neural networks while maintaining or even improving model performance. Due to reduced parameter precision and high number of zero-elements, the rendered networks are highly compressible in terms of file size, up to $103\times$ compared to the full-precision unquantized DNN model. Our approach was evaluated on different types of models and datasets (including Google Speech Commands, CIFAR-10 and Pascal VOC) and compared with previous work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑