arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.01604cs.CVcs.AIcs.LG
收藏

超越分类精度的模型压缩

Going Beyond Classification Accuracy Metrics in Model Compression

NVIDIA(英伟达);TU Kaiserslautern(凯泽斯劳滕工业大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);University of Utah(犹他大学)

Vinu Joseph, Shoaib Ahmed Siddiqui, Aditya Bhaskara, Ganesh Gopalakrishnan, Saurav Muralidharan, Michael Garland, Sheraz Ahmed, Andreas Dengel

摘要:

随着边缘计算设备的兴起,部署节能和资源高效模型的需求越来越大。大量的研究致力于开发方法,可以大大减少模型的大小,而不影响标准指标,如top-1的准确性。然而,这些修剪方法往往会导致其他指标(如跨类公平性和可解释性)的显著不匹配。为了克服这种偏差,我们提出了一种新的多部分损失函数的启发知识蒸馏文献。通过大量的实验,我们证明了我们的方法在不同的压缩算法,架构,任务以及数据集的有效性。特别是,我们获得了压缩模型和参考模型之间的预测不匹配数量的最多$4.1\times$减少,以及在参考模型做出正确预测的情况下最多$5.7\times$减少;所有这些都不改变压缩算法,并且对损失函数进行微小修改。此外,我们还展示了如何在模型的预测之间引入简单的对齐,从而自然地提高了其他指标的对齐,包括公平性和属性。因此,我们的框架可以作为一个简单的即插即用组件的压缩算法在未来。

英文摘要:

With the rise in edge-computing devices, there has been an increasing demand to deploy energy and resource-efficient models. A large body of research has been devoted to developing methods that can reduce the size of the model considerably without affecting the standard metrics such as top-1 accuracy. However, these pruning approaches tend to result in a significant mismatch in other metrics such as fairness across classes and explainability. To combat such misalignment, we propose a novel multi-part loss function inspired by the knowledge-distillation literature. Through extensive experiments, we demonstrate the effectiveness of our approach across different compression algorithms, architectures, tasks as well as datasets. In particular, we obtain up to $4.1\times$ reduction in the number of prediction mismatches between the compressed and reference models, and up to $5.7\times$ in cases where the reference model makes the correct prediction; all while making no changes to the compression algorithm, and minor modifications to the loss function. Furthermore, we demonstrate how inducing simple alignment between the predictions of the models naturally improves the alignment on other metrics including fairness and attributions. Our framework can thus serve as a simple plug-and-play component for compression algorithms in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑