arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1912.01274cs.LGcs.CVstat.ML
收藏

无数据模型压缩方法

The Knowledge Within: Methods for Data-Free Model Compression

Habana Labs Research(Habana Labs研究院);Technion(以色列理工学院)

Matan Haroush, Itay Hubara, Elad Hoffer, Daniel Soudry

摘要:

最近,大量的研究集中在压缩和加速深度神经网络(DNN)上。到目前为止,高压缩率算法需要部分训练数据集进行低精度校准或微调过程。然而,当数据不可用或包含敏感信息时,例如在医疗和生物识别用例中,此要求是不可接受的。我们提出了三种从训练模型生成合成样本的方法。然后,我们演示了如何使用这些样本来校准和微调量化模型,而无需在此过程中使用任何真实数据。与原始训练集相比,我们表现最好的方法具有可忽略不计的准确性下降。该方法利用训练模型的内在批量归一化层的统计信息,可用于评估数据相似性。我们的方法开辟了一条通往真正的无数据模型压缩的道路,减轻了模型部署过程中对训练数据的需求。

英文摘要:

Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑