arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1901.09421stat.MLcs.ITcs.LGmath.IT
收藏

基于模型压缩的人口风险改善的信息论理解

Information-Theoretic Understanding of Population Risk Improvement with Model Compression

University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University at Buffalo, the State University of New York(纽约州立大学布法罗分校)

Yuheng Bu, Weihao Gao, Shaofeng Zou, Venugopal V. Veeravalli

摘要:

我们通过研究模型压缩下泛化误差的降低和经验风险的增加之间的权衡,证明了模型压缩可以提高预训练模型的群体风险。我们首先证明,模型压缩减少了信息理论界的泛化误差,这允许解释模型压缩作为一种正则化技术,以避免过度拟合。然后,我们用率失真理论描述了模型压缩的经验风险的增加。这些结果意味着,如果泛化误差的减少超过经验风险的增加,则模型压缩可以改善总体风险。我们通过一个线性回归的例子表明,由于模型压缩,这种人口风险的降低确实是可能的。我们的理论结果进一步表明,Hessian加权$K$ -均值聚类压缩方法可以通过正则化聚类中心之间的距离来改进。我们提供了神经网络的实验来支持我们的理论主张。

英文摘要:

We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalization error; this allows for an interpretation of model compression as a regularization technique to avoid overfitting. We then characterize the increase in empirical risk with model compression using rate distortion theory. These results imply that the population risk could be improved by model compression if the decrease in generalization error exceeds the increase in empirical risk. We show through a linear regression example that such a decrease in population risk due to model compression is indeed possible. Our theoretical results further suggest that the Hessian-weighted $K$-means clustering compression approach can be improved by regularizing the distance between the clustering centers. We provide experiments with neural networks to support our theoretical assertions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑