arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.07668cs.LG
收藏

基于深度假设的鲁棒模型压缩

Robust Model Compression Using Deep Hypotheses

Tel Aviv University(特拉维夫大学)

Omri Armstrong, Ran Gilad-Bachrach

摘要:

理想情况下,机器学习模型应该是紧凑和健壮的。紧凑性提供效率和可理解性,而健壮性提供弹性。近年来,这两个主题都得到了研究,但都是孤立的。在这里,我们提出了一个强大的模型压缩计划,这是独立的模型类型:它可以压缩集成,神经网络和其他类型的模型到不同类型的小模型。主要的组成部分是从稳健统计学中得出的深度概念。最初,深度被引入作为样本中点的中心性的度量,使得中值是最深的点。这个概念被扩展到分类功能,这使得有可能定义的深度假设和中位数假设。已经提出了近似中位数的算法,但它们仅限于二进制分类。在这项研究中,我们提出了一种新的算法,多类经验中值优化(MEMO)算法,发现一个深的假设,在多类任务,并证明其正确性。这导致我们的紧凑的鲁棒估计中值置信优化(CREMBO)算法的鲁棒模型压缩。我们通过将神经网络和随机森林压缩成可解释模型的小决策树,证明了该算法的成功经验,并表明它们比其他可比方法更准确和鲁棒。此外,我们的实证研究表明,我们的方法优于知识蒸馏DNN到DNN压缩。

英文摘要:

Machine Learning models should ideally be compact and robust. Compactness provides efficiency and comprehensibility whereas robustness provides resilience. Both topics have been studied in recent years but in isolation. Here we present a robust model compression scheme which is independent of model types: it can compress ensembles, neural networks and other types of models into diverse types of small models. The main building block is the notion of depth derived from robust statistics. Originally, depth was introduced as a measure of the centrality of a point in a sample such that the median is the deepest point. This concept was extended to classification functions which makes it possible to define the depth of a hypothesis and the median hypothesis. Algorithms have been suggested to approximate the median but they have been limited to binary classification. In this study, we present a new algorithm, the Multiclass Empirical Median Optimization (MEMO) algorithm that finds a deep hypothesis in multi-class tasks, and prove its correctness. This leads to our Compact Robust Estimated Median Belief Optimization (CREMBO) algorithm for robust model compression. We demonstrate the success of this algorithm empirically by compressing neural networks and random forests into small decision trees, which are interpretable models, and show that they are more accurate and robust than other comparable methods. In addition, our empirical study shows that our method outperforms Knowledge Distillation on DNN to DNN compression.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑